SAINTCON 26 · NetOps
NOC · OPSSpaceways · Flight 26
The network,
by the numbers
by the numbers
Three days of SAINTCON on the wire and in the air.
Network byAristaInternet byCentracom
Before the doors opened · Saturday
The 19 minute 55 second ghost
SymptomInternet gone for 25-38 seconds, every 19:55. Inside the network, zero loss.
Ruled outCabling, optics, load, DHCP, config pushes, a dashboard bug. A timer is software, not a cable.
Found itAsked the network who owns our public IP. Two machines answered. A forgotten firewall upstream had our address.
FixNew IP at 1:31 PM. The next two windows: clean. Then a permanent alarm on it.
source: ping logger + SNMP + ARP capture, Oct 24
At a glance
Tuesday through Thursday
Wi-Fi devices at once
0
Peak, Wed 11:30 AM
On a single access point
0
AP 2-03, AI Maker Corner, Wed 9:49 AM
Internet burst
0Gbps
Wed 12:31 PM, on a 10 Gbps Centracom fiber
Downloaded from the internet
0TB
plus 0.95 TB up, counted since Wed 11:40 AM
Access points
0
Arista C-460 Wi-Fi 6E, 2.4 + 5 + 6 GHz
Network devices polled live
0
Gateway + switches over SNMP every few seconds, drawn on a 3D map of the venue
source: netdash (CV-CUE + SNMP), Thu Oct 29, 3:00 PM MDT
Timelapse · Wi-Fi by access point
Three days on the Wi-Fi
Mountain time
TUE06:00
Devices on Wi-Fi
0
Busiest right now
source: CV-CUE client counts per AP via netdash, 5 min samples · 5-2425-4950-7980+
Timelapse · whole venue
Every day has a heartbeat
source: CV-CUE associated clients, all APs, 5 min samples
Timelapse · where the crowd goes
Follow the crowd
source: CV-CUE clients per AP, grouped by nearest floor-plan area, 08:00-18:00 each day
Timelapse · the internet link
The pipe
Mountain time
WED11:40
Downloaded so far
0.00 TB
Uploaded so far
0.00 TB
Right now of 10 Gbps
source: SNMP on the gateway's Centracom port via netdash wall board, sampled every ~3 min; burst = fastest single poll
Wi-Fi tuning, mid-con
Same crowd, better spread
Level 2 during the morning keynote, devices per access point. Wednesday vs Thursday.
source: CV-CUE clients per AP, Wed 9:45 AM and Thu 9:30 AM samples
Who was on the air · Thu 12:23 PM census
You brought good radios
Band right now · 942 devices
5 GHz · 533
6 GHz · 295
MLO · 104
2.4 GHz: 10 devices. MLO = Wi-Fi 7 multi-link, using more than one band at once.
96%
of devices on 5 GHz said they could also do 6 GHz (509 of 533). Next year's easy win: steer them up.
35 of 36
reporting access points ran a 6 GHz radio. 5 GHz at 40 MHz channels, 6 GHz at 80 MHz.
89%
Wi-Fi 6 or newer
97%
WPA3 (632 of 651 that reported security)
6
devices old enough to be 2.4 GHz b/g only
103
Wi-Fi 7 devices
-57.5 dBm
median signal. Only 5.6% of devices were under -75 dBm at Thursday's first weak-signal poll.
source: CV-CUE client records, counted in aggregate only (no device list kept)
The fabric
What carried it
source: CloudVision inventory, venue map, netdash SNMP
What we built
A live 3D map of the whole venue
Tour mode, recorded on the demo feed · map.saintnoc.org (team) · kiosk.saintnoc.org (public)
What we built
The boards on the wall
NOC board · links by load, every switch, Wi-Fi, DHCP, services, down alarms (addresses blurred)
Click any screen to blow it up. When something drops, the NOC plays an "uh oh" three times and Skippy shows up to blame someone.
Wi-Fi board · hot APs, con records, heat strip
Density map · replay any minute of the day
Built this week
Ten things that did not exist a month ago
3D venue map
map.saintnoc.org
Live links, switches, APs, a tour mode and a spaceship.
Public kiosk
kiosk.saintnoc.org
The attendee view: crowds and links, nothing internal.
NOC board
noc.saintnoc.org
Every link and switch, plus an alarm you can hear.
Wall board
wall.saintnoc.org
Crowd, internet and backbone for the room screens.
Wi-Fi board
wifi.saintnoc.org
Hot APs, records, weak signal by network.
LED panel
NOC wall
32×8 pixels of live ticker, streamed at 21 fps.
Badge field map
SAINTCON 2024 badge
Reflashed into a handheld map of what is down.
Walk test
map, phone in hand
Which AP grabs you where you stand.
Uh oh
NOC alarm
Something drops, the room hears it. Three times.
Skippy
on every board
The team bot, animated by Klipper, judging us.
Network by Arista, internet by Centracom
38 access points, a two-spine switch fabric, a 10 Gbps fiber, and a Starlink dish on the balcony just in case.
NetOps · SAINTCON 26
Thank you
L34NPinballKenpoSamuraiMiLwOrkZGaming On The FloorshiftyAristaSkippy
and Centracom for the 10 Gbps pipe.