Which day should we drive? Watching border queues with Home Assistant and a vision model

Twice a year we drive home. Same route, same border crossings, same question in the queue: should we have left a day earlier?

Nobody knows. I wanted to know which day and hour were best at each crossing. A navigation app only shows the queue right now, so I decided to record it myself.

Public traffic cams exist for the crossings on our route, and I already had Home Assistant running. So I made it photograph every camera on a schedule, ask a vision model how bad the queue looks, and keep the answer as a sensor.

This post covers the idea, the build, and what I'd do differently.

Contents

• Overview
  • The idea
  • What you end up with
  • Tech stack
• The setup
  • Step 1: The cameras
  • Step 2: Snapshot and check the image
  • Step 3: Ask the model
  • Step 4: Store it so you can chart it
  • Step 5: Only run when it matters
• Running it
  • What it costs
  • Testing
  • Troubleshooting
• Retrospective
  • What I learned
  • What I would change

Overview

The idea

Peak travel follows the school calendar. In countries where school holidays are staggered by region (Germany is the textbook case), traffic comes in waves.

Outbound, traffic usually starts on the Friday before the first region's holidays begin. On the way back, it usually starts building about a week before the first region's holidays end.

So each direction gets its own collection window. Outside the windows I don't need data. Inside them I want a reading every 30 minutes from every camera on the route. Then I look at the graphs and pick a day.

route:   origin ──► crossing A ──► crossing B ──► home
                       │              │
                  cameras per     cameras per
                    direction       direction

Each camera watches one direction of one crossing. That turns out to matter.

What you end up with

every 30 min, daylight only, only inside the collection windows
        │
        ▼
 for each camera:
   snapshot ──► JPEG on disk (/media/traffic/<camera>/<timestamp>.jpg)
        │
   is it a real image?  no ──► retry once ──► still no ──► "no data"
        │ yes
        ▼
   vision model ──► { congestion 0-4, image_kind, note, overlay_time }
        │
   image_kind != live?  ──► "no data"
        │
        ▼
   helpers + one sensor per camera ──► history graph, per direction

Each camera gets a sensor with a 0 to 4 score, the crossing and direction as attributes, and a note like "long queue of heavy trucks along the left side". Two history graphs show every crossing for one direction on one time axis.

I have 13 cameras and one image each per run. No GPU involved.

Tech stack

The setup

Step 1: The cameras

I used a public site that lists traffic cams for border crossings. Each camera there has a still-image URL.

Add each one in Home Assistant as a Generic Camera:

Settings → Devices & services → Add integration → Generic Camera

Use the still image URL from the site. You don't need a stream URL, we only want stills. Pick a boring, consistent naming pattern, because the name becomes the entity id and everything else is derived from it.

Put all the cameras in one area (mine is called Traffic Cams). The automation loops over the area, so adding a camera later is a UI-only change.

Gotcha: the entity id comes from the name you type when you add the camera, and the friendly name you see afterwards can look different. Check the id in Settings → Entities after adding each one. If you rename a camera later, everything keyed on the old id stops working, without an error.

Verify it worked: in Developer tools → Template:

{{ area_entities('traffic_cams') | select('match', 'camera\.') | list | length }}

It should print your camera count.

Step 2: Snapshot and check the image

camera.snapshot fetches the still and writes it to disk. Home Assistant only writes to allow-listed folders, so /media has to be on the list. On my HAOS install it already was.

- action: camera.snapshot
  target:
    entity_id: "{{ cam }}"
  data:
    filename: "/media/traffic/{{ slug }}/{{ now().strftime('%Y-%m-%d_%H%M') }}.jpg"

The file name carries the camera and the timestamp. That's the whole "label the images" requirement, and it keeps the archive browsable without a database.

Next, check that you actually got an image.

A snapshot can "succeed" and still save an error page as .jpg. In my first audit, 1 fetch out of 65 came back as an HTTP 500 with a 26-byte text body. Left alone, it would have gone straight to the model.

Home Assistant templates can't look inside files, so I used a tiny shell_command:

shell_command:
  # prints "ok" only if the file exists, is >= 5000 bytes, and starts with the JPEG magic bytes
  traffic_cams_probe: >-
    sh -c 'f="{{ path }}"; [ -f "$f" ] && [ "$(wc -c < "$f")" -ge 5000 ] &&
    [ "$(head -c 3 "$f" | od -An -tx1 | tr -d " \n")" = "ffd8ff" ] && echo ok'

If the probe fails, take the snapshot once more. If it fails again, skip the model call and record "no data".

Gotcha: a templated shell_command runs without a shell, so pipes and && do nothing unless you wrap everything in sh -c. Check that wc, head, od and tr exist in the Home Assistant container before you trust the probe.

Verify it worked: run the script by hand and open Media → My media. You should see one folder per camera.

Step 3: Ask the model

Home Assistant has an ai_task.generate_data action. It takes an image attachment and a structure, so you get typed fields back instead of a paragraph.

- action: ai_task.generate_data
  data:
    entity_id: ai_task.my_vision_model
    task_name: traffic
    instructions: >-
      This is a still from a border-crossing traffic camera. Rate the queue
      from 0 (empty, free flow) to 4 (standstill, long queue).
      If the image is not a live road view (poster, offline, black, test
      pattern) say so in image_kind.
    structure:
      congestion:
        selector: { number: { min: 0, max: 4 } }
        required: true
      image_kind:
        selector: { select: { options: [live, poster, offline, night, unclear] } }
        required: true
      note:
        selector: { text: }
      overlay_time:
        selector: { text: }
    attachments:
      media_content_id: "media-source://media_source/local/traffic/{{ slug }}/{{ stamp }}.jpg"
      media_content_type: image/jpeg
  response_variable: result

I added two fields late, and I'd now keep them in any version of this.

image_kind makes the model say whether it's looking at a real road. A camera could show a poster, an "offline" tile or a black frame, and a model could rate that "0, empty road". That would put a false zero in the series. My cameras were all live when I checked, but I didn't want the pipeline to depend on that.

overlay_time is the clock most cameras burn into the frame. The model reads it, and later you can compare it with the real time to catch frozen or delayed feeds.

For the model I used a small, cheap one through OpenRouter, which Home Assistant has a core integration for. You add the integration, then an AI task sub-entry with the model you want. Swapping models later is one dropdown.

The rule I stuck to: the model reports, the code decides.

image_kind != live  →  congestion = -1  (no data)
live                →  congestion clamped to 0..4

That gate is plain template logic and I test it offline. The model doesn't decide whether a reading is stored.

Verify it worked: run the script once. Each camera's note should match what you see when you open its JPEG.

Step 4: Store it so you can chart it

Per camera I write four helpers: congestion (input_number, range -1 to 4), plus note, image_kind and overlay_time (all input_text).

The helpers are only the write target. What you chart is a template sensor on top:

template:
  - sensor:
      - name: "Crossing A south→north"
        unique_id: traffic_crossing_a_sn
        state_class: measurement          # long-term statistics, so you can export a CSV later
        state: >-
          {% set v = states('input_number.traffic_crossing_a_sn_congestion') | float(-1) %}
          {{ v if v >= 0 else none }}
        attributes:
          crossing: Crossing A
          direction_group: southbound
          image_kind: "{{ states('input_text.traffic_crossing_a_sn_image_kind') }}"

-1 becomes none, so the history graph shows a gap instead of a fake zero. state_class: measurement makes Home Assistant keep long-term statistics, which outlive the 10-day recorder default and export to CSV.

Then two history-graph cards give me what I wanted in the first place, one per travel direction with a line per crossing:

type: history-graph
title: Traffic southbound
hours_to_show: 72
entities:
  - sensor.traffic_crossing_a_sn
  - sensor.traffic_crossing_b_sn

Gotcha: helpers defined in YAML can't be grouped under a device, because only integrations own devices. I wanted a tidy "Traffic" device and ended up with a naming prefix and a dashboard. If you need a real device, MQTT discovery gives you one, but you have to run a broker.

Gotcha: dashboards have URL paths, and Home Assistant insists on a hyphen in them. traffic-cams works, traffic doesn't.

Verify it worked: open the dashboard after a few runs. Gaps in a line are fine. A flat zero for a crossing you know is busy is not.

Step 5: Only run when it matters

13 cameras every 30 minutes, all year, adds cost and fills the disk for no benefit. The traffic I care about lives in two or three windows a year.

So the windows sit in one list at the top of the script:

windows:
  - name: christmas
    start: "2030-12-21"
    end:   "2031-01-08"
    directions: [southbound, northbound]
  - name: summer_southbound
    start: "2031-06-24"          # the Friday before the first region's holidays start
    end:   "2031-08-02"          # the day we drive
    directions: [southbound]
  - name: summer_northbound
    start: "2031-07-29"          # a week before the first region's holidays end
    end:   "2031-09-13"          # school starts again in the last region
    directions: [northbound]

Every camera carries a direction. A camera is processed only if today falls inside a window that lists its direction. The date is now().date(), so it follows the Home Assistant timezone, not UTC.

The two summer windows overlap for a few days. During the overlap both directions run.

One window per direction roughly halved the number of model calls compared with "run everything all summer". The cameras facing the other way would have recorded weeks of data I don't need.

The automation itself stays dumb:

automation:
  - id: traffic_cams_daylight
    alias: Traffic cams (daylight)
    triggers:
      - trigger: time_pattern
        minutes: "/30"
    conditions:
      - condition: state
        entity_id: sun.sun
        state: above_horizon
    actions:
      - action: script.traffic_cams_run
    mode: single
    initial_state: true

The sun condition is there because I decided night frames aren't worth rating.

The script also has an ignore_windows field. It exists for testing. Outside every window the script correctly does nothing, and you can't prove that without a way to make it do something.

Gotcha: initial_state is re-applied on every Home Assistant restart. A toggle in the UI won't survive it. Whatever the YAML says wins.

Gotcha: an automation's entity id comes from its id. I renamed mine once, because the first name had the interval in it, and the old entity hung around as an unavailable orphan until I deleted it from the entity registry. Don't put anything in an id that might change.

Verify it worked: run the script with ignore_windows: false on a day outside every window. No new files should appear. Run it again with true and every camera should produce one.

Running it

What it costs

I measured, then extrapolated. All my testing cost $0.04 in total. I didn't count the calls exactly, roughly 85, so call it $0.0005 per call. One run is one call per camera.

That's about $11 for a whole season. Put a credit limit on the API key at roughly double that, so a bug can't turn into a bill.

Going from every 20 to every 30 minutes cut the cost by a third, so I took the saving.

Storage is small. The JPEGs are 15 to 55 KB each, around 640 MB for the season. I don't delete anything. The images are the raw data if I ever want to re-score them with a better model.

Testing

Everything deterministic gets tested without Home Assistant. The test script renders the real templates from the YAML with Jinja2, so it exercises the same code and not a copy of it.

It checks that:

• the YAML parses with no duplicate keys,
• the expected helpers exist for every camera,
• scores are clamped (-3 → 0, 7 → 4, abc → 0),
• the image gate works (non-live becomes -1, an unknown kind becomes unclear and then -1),
• the window gate behaves on the edge days: day before, first day, last day, day after, the overlap, and a random day in November,
• the camera table in the README matches what the generator prints.

That's about 300 checks.

Then there's a live scenario against the real instance. It runs the script gated and expects nothing, runs it forced and expects one image and one fresh sensor per camera, and checks the automation exists.

Gotcha: my first live test failed for a camera whose note was word for word the same as the previous run. Home Assistant doesn't bump last_updated when a value doesn't change, and an empty road looks the same every time. Test "did it run" with something that can't repeat, like the image file.

Troubleshooting

Rows marked * I haven't hit myself. They follow from how the pieces work.

On that last row: Home Assistant OS has three names for the same config folder:

Retrospective

What I learned

1. A successful fetch isn't proof of a good image. One of my 65 fetches returned an error page. I also ran a 12-minute audit of the cameras, five rounds three minutes apart. Every image was different, which shows the snapshots come from live broadcasts, and every one was a real scene. But one camera's burned-in clock was 26 minutes behind, another's was two hours ahead, and one camera was mounted upside down. Stale or poster frames didn't show up, but nothing in the pipeline would have caught them if they had.

2. Let the model describe and let the code decide. The model says what it sees, and plain code applies the thresholds. That kept the rules testable, and swapping the model touched no logic.

3. Direction is the unit of analysis. My first grouping was "into the country / out of the country", and it was wrong for what I needed. What matters is which way you're travelling along the route, because a crossing has cameras for both directions, and their traffic differs. Splitting by southbound and northbound fixed that, and it gave me the per-direction windows for free.

4. Limit when you collect before you scale how much. Most of the cost saving came from not collecting, and the windows were a small change.

5. A sentinel beats a fake zero. -1 for "no data" plus none in the sensor gives you gaps in the chart. A made-up 0 puts false data in the chart.

6. Check the system before you document it. At one point I had documentation naming the wrong AI provider. The model's display name looked like it came from one vendor but was served through another. The answer was one API call away (the list of loaded components).

7. Test the bypass, not just the gate. If your automation is correctly silent most of the year, you need a switch that makes it run, or you can never show that it works.

What I would change

Starting over, knowing what I know now:

1. I'd run the freshness audit on day one. Five fetches per camera, three minutes apart, compare hashes, look at the images. A short audit that would have shaped the design earlier: the gate, the image_kind field, the retry.
2. I'd store the model and prompt version with every reading. Right now I keep the answer but not which model and prompt produced it. If I change either mid-season, the series has a silent seam in it, and a version field costs nothing.
3. I'd add a staleness check outside Home Assistant. It can't hash a file. A small script that compares each image with the previous one (a perceptual hash) and flags "unchanged for N runs" would catch frozen feeds that the model still calls live. I keep every image, so I can do this after the fact, but I'd rather have it in the loop.
4. I'd calibrate the score. "0 to 4" comes from a prompt, not from data, and a single frame shows density, not speed. Before trusting the numbers I'd label a few dozen frames by hand and compare them with the model's scores.
5. I'd set the windows from data, not from a calendar. Mine come from holiday dates plus a feel for when people start moving. A cheap all-year run, say every 3 hours, would show when the waves really start.
6. I'd put the three folder names on the first page of the README. It got in the way once.
7. I'd keep anything that might change out of names. Intervals, dates and providers don't belong in an entity id.
8. I'd look at pictures earlier. I built most of the pipeline before I'd looked at enough frames to see what the data would really be like: low resolution, hazy, one camera rotated. A hundred frames in a folder tell you things a design doc won't.

The package, tests and docs live in my homelab repo. If you want the real YAML instead of the trimmed versions above, ask and I'll publish a cleaned copy.