I'm doing a student project on NYC's facade inspection program. Buildings over six stories get rated Safe, SWARMP or Unsafe every five years, and a 2024 law requires the city to use a predictive model to prioritize inspections. I couldn't find a public one, so I tried building my own from the public filings.
Task: predict whether a building will be rated Unsafe at its next inspection cycle.
Setup: trained on cycles 6→7 and 7→8, tested once on 8→9 (about 11k buildings, 25% Unsafe).
Results on the held-out cycle:
- Top 20% of the ranked list catches 36.4% of Unsafe buildings (random order: 19.8%, last rating only: 32.6%)
- LightGBM and logistic regression perform the same
- A random split of the same data gave 46.8%, so about 10 points too optimistic
- Each cycle is harder to predict than the last
- A conformal recall guarantee works (90% target gave 89.2%) but needs about 75% of buildings inspected
|
Random split |
Time split (test 8→9) |
| Recall in top 20% |
46.8% |
36.4% |
| PR-AUC |
0.548 |
0.430 |
| AUC |
0.763 |
0.687 |
| Recall target |
Realized on 8→9 |
Share of buildings to inspect |
| 80% |
80.6% |
62.6% |
| 90% |
89.2% |
75.4% |
| 95% |
93.6% |
83.3% |
Same-cycle CV recall in top 20% (a rough ceiling): 6→7 53.3%, 7→8 47.7%, 8→9 39.2%
After I looked at 8→9, I split it into dev and reserve halves and only kept ideas that helped both. None did.
Questions:
1. Is that dev/reserve approach reasonable once a test set has been used?
2. What would you use for conformal calibration when labeling rules change between cycles?
3. Labels come from inspectors hired by the buildings themselves, not by the city. Any ideas for handling that with public data?
Data is NYC Open Data (DOB NOW facade filings). Numbers are in the tables above.