From a CSV file to a dataframe

TL;DR#

Tonight we sent data to the course API with a POST, saved every town in Massachusetts to a CSV file, read the file back, and answered one question with a loop, in polars and in SQL. The habit worth keeping: everything you read back from a CSV file is text until you convert it. int() comes before any comparison or sum.

What we covered#

A 200 means the server answered. The misspelled Wellesly returns 200 and a reply with nothing in it but generationtime_ms. Read the reply as well as the status code.

len counts what a value is made of: the characters of a string, the items of a list, the key-value pairs of a dictionary. len(reply["data"][0]) is 3, because reply["data"][0] is one person's dictionary, with three keys.

Read a nested reply one layer at a time. reply["data"] is the list, [1] the second person in it, and ["age"] that person's age: reply["data"][1]["age"].

A function that prints gives back None. print puts the label on the screen and the value goes nowhere, so result = make_label("Wellesley", "Norfolk") stores None. Change print(label) to return label and result holds the text. Full reference: Python §8, Functions. An analogy: Print vs Return, and the comment under it about counting steps.

Abstraction is using a function by what it takes and returns, without reading its inside, as you use requests.get. Decomposition is breaking a problem into pieces small enough to write.

A GET puts its parameters in the address, after ?. A search form builds that address for you: Google's search box sends /search?q=babson, Amazon's sends /s?k=babson, and params={"county": "Norfolk"} sends /ma/towns?county=Norfolk. A browser's address bar sends only GET requests.

A POST carries its data in the body. A login form uses POST, so your password never appears in the address bar. On the board, json= puts your message in the body and the x-token header says who sent it.

A saved file runs again without the network. An API can change, limit how often you ask, or charge for each request. A copy on disk gives the same answer next week, and Mini Project 2 asks you to save what you fetched.

A CSV file has no types. csv.DictReader gives every value back as a string, so town["average_single_family_value_dollars"] >= 1000000 stops with TypeError: '>=' not supported between instances of 'str' and 'int'. Wrap the value in int() first.

Count with a dictionary. The first time a county appears, start it at zero, then add one. counts.get(county, 0) does both in a single line: counts[county] = counts.get(county, 0) + 1.

A count can mislead. Middlesex has the most million-dollar towns, 14, and it also has 54 towns in all. By share, Dukes and Nantucket come first, with every town worth $1,000,000 or more, and Middlesex is at 26%. The chart in section 4 draws the share.

A loop, polars and SQL give the same counts. In polars, .sort("towns", descending=True) puts the largest first; Python's own sorted() calls the same option reverse=True.

A CSV file opens in VS Code. The extension Rainbow CSV colors each column so the file is easier to read.

Running a script every morning#

One of tonight's talks used GitHub Actions to run a Python script on a schedule and send the results to a Telegram bot. What came up in the discussion:

• A workflow is a file in .github/workflows/. on: schedule: with a cron: line runs your script on GitHub's computers. Times are in UTC unless you add a timezone:, runs can start late near the top of the hour, and the shortest interval is five minutes. Reference: GitHub's documentation for schedule
• In a public repository, a schedule stops after 60 days with no activity. A commit starts it again
• A token goes in a secret, never in a file: Settings → Secrets and variables → Actions. A Telegram bot's token comes from BotFather, and sending a message also needs the chat's id
• Give a project like this its own repository, separate from the course repository. It can be private
• Read a site's terms before you collect from it. LinkedIn forbids collecting its listings with a script, so the bot reads companies' public career pages instead

Letting an agent decide#

The second talk asked whether you would follow an AI's advice to close a position at a loss. Meta's Muse is an agent that acts for you. It connects to Gmail and Google Calendar and needs your approval before it sends a message or spends money. Each connection trades some privacy for help. If you let an agent make decisions, keep a record of every time you overrode it, so that you can check later which of you was right.

If your environment stopped working#

• Your repository is in OneDrive or iCloud Drive. Deleting .venv and running uv sync works until the next sync. Move the repository out: FAQ, "My repository is in iCloud Drive or OneDrive"
• uv stops with os error 4551 on Windows. That is Smart App Control: FAQ, "uv stops with os error 4551"

Public links#

• Session page, with the slides and notebook 5
• Mini Project 2
• Course API documentation
• Python §8, Functions
• FAQ

Before next session#

1. Mini Project 2 is due Sunday 10/25. Submit the link to your notebook inside projects/mp2/. The instructions are on the Mini Project 2 page.
2. Notebook 4, section 4, 04-functions-and-apis.py.
3. Notebook 5, sections 1 to 4, 05-csv-and-dataframes.py: A to D.

Checkpoint 3 is due Sunday 10/25. The full checklist is on Canvas.