Input and files
Contents
Input and files#
Very often, we want to work with data that has already been collected or generated elsewhere. This might take the form of data from online surveys, or acoustic monitoring sensors, or perhaps the results of intensive fluid-dynamics simulations carried out on huge remote clusters of computers. There are a vast number of possible formats for data, and ways of reading it into Python, so we will focus only on the simplest.
CSV files#
A comma-separated value (csv) file contains the data in a simple form: each line of the file contains a number of values, separated by commas. Often the first line consists of headers (i.e. bits of text describing each column of data).
As an example, the file data/strathkiness-rainfall.csv consists of daily rainfall data for Strathkinness (near St Andrews), between August 1st 2020 and August 23rd 2021. This data is published by SEPA under the Open Government License v3.0; you can find the most recent rainfall data here. The first few lines of that file look like this:
Timestamp,Value
01/08/2020 09:00:00,0.2
02/08/2020 09:00:00,0.0
03/08/2020 09:00:00,2.0
This has two columns: Timestamp, the day the data is for, and Value, the total daily rainfall (in millimetres).
Using csv:#
We could read the rainfall data into Python like so:
# we can use the csv module to read csv files
import csv
#
with open('data/strathkiness-rainfall.csv', newline='') as csvfile:
# Create a "reader" object to do the reading.
# You *can* use things other than commas in csv files, so we have to
# tell the reader that the value separator is a comma here.
reader = csv.reader(csvfile, delimiter=',')
rain_data = list(reader) # convert the data to a list
rain_data
[['Timestamp', 'Value'],
['01/08/2020 09:00:00', '0.2'],
['02/08/2020 09:00:00', '0.0'],
['03/08/2020 09:00:00', '2.0'],
['04/08/2020 09:00:00', '15.8'],
['05/08/2020 09:00:00', '0.4'],
['06/08/2020 09:00:00', '0.0'],
['07/08/2020 09:00:00', '0.0'],
['08/08/2020 09:00:00', '0.0'],
['09/08/2020 09:00:00', '0.0'],
['10/08/2020 09:00:00', '1.6'],
['11/08/2020 09:00:00', '7.6'],
['12/08/2020 09:00:00', '0.0'],
['13/08/2020 09:00:00', '0.8'],
['14/08/2020 09:00:00', '0.0'],
['15/08/2020 09:00:00', '0.0'],
['16/08/2020 09:00:00', '0.2'],
['17/08/2020 09:00:00', '2.2'],
['18/08/2020 09:00:00', '0.0'],
['19/08/2020 09:00:00', '4.4'],
['20/08/2020 09:00:00', '1.4'],
['21/08/2020 09:00:00', '5.0'],
['22/08/2020 09:00:00', '0.0'],
['23/08/2020 09:00:00', '1.0'],
['24/08/2020 09:00:00', '3.2'],
['25/08/2020 09:00:00', '41.4'],
['26/08/2020 09:00:00', '1.8'],
['27/08/2020 09:00:00', '15.2'],
['28/08/2020 09:00:00', '0.0'],
['29/08/2020 09:00:00', '0.0'],
['30/08/2020 09:00:00', '0.0'],
['31/08/2020 09:00:00', '0.0'],
['01/09/2020 09:00:00', '0.0'],
['02/09/2020 09:00:00', '8.4'],
['03/09/2020 09:00:00', '0.2'],
['04/09/2020 09:00:00', '1.2'],
['05/09/2020 09:00:00', '0.0'],
['06/09/2020 09:00:00', '3.0'],
['07/09/2020 09:00:00', '0.0'],
['08/09/2020 09:00:00', '2.0'],
['09/09/2020 09:00:00', '0.0'],
['10/09/2020 09:00:00', '0.0'],
['11/09/2020 09:00:00', '0.4'],
['12/09/2020 09:00:00', '2.2'],
['13/09/2020 09:00:00', '0.0'],
['14/09/2020 09:00:00', '0.0'],
['15/09/2020 09:00:00', '0.2'],
['16/09/2020 09:00:00', '0.0'],
['17/09/2020 09:00:00', '0.0'],
['18/09/2020 09:00:00', '0.2'],
['19/09/2020 09:00:00', '0.6'],
['20/09/2020 09:00:00', '0.2'],
['21/09/2020 09:00:00', '0.0'],
['22/09/2020 09:00:00', '0.6'],
['23/09/2020 09:00:00', '0.0'],
['24/09/2020 09:00:00', '0.0'],
['25/09/2020 09:00:00', '0.0'],
['26/09/2020 09:00:00', '0.2'],
['27/09/2020 09:00:00', '0.0'],
['28/09/2020 09:00:00', '0.0'],
['29/09/2020 09:00:00', '3.8'],
['30/09/2020 09:00:00', '18.2'],
['01/10/2020 09:00:00', '1.2'],
['02/10/2020 09:00:00', '6.4'],
['03/10/2020 09:00:00', '65.8'],
['04/10/2020 09:00:00', '8.4'],
['05/10/2020 09:00:00', '3.2'],
['06/10/2020 09:00:00', '0.0'],
['07/10/2020 09:00:00', '0.0'],
['08/10/2020 09:00:00', '0.0'],
['09/10/2020 09:00:00', '1.0'],
['10/10/2020 09:00:00', '0.4'],
['11/10/2020 09:00:00', '3.6'],
['12/10/2020 09:00:00', '4.6'],
['13/10/2020 09:00:00', '7.2'],
['14/10/2020 09:00:00', '0.0'],
['15/10/2020 09:00:00', '0.0'],
['16/10/2020 09:00:00', '0.0'],
['17/10/2020 09:00:00', '0.0'],
['18/10/2020 09:00:00', '1.8'],
['19/10/2020 09:00:00', '11.0'],
['20/10/2020 09:00:00', '1.2'],
['21/10/2020 09:00:00', '6.4'],
['22/10/2020 09:00:00', '3.4'],
['23/10/2020 09:00:00', '3.2'],
['24/10/2020 09:00:00', '0.6'],
['25/10/2020 09:00:00', '0.2'],
['26/10/2020 09:00:00', '3.6'],
['27/10/2020 09:00:00', '5.2'],
['28/10/2020 09:00:00', '0.0'],
['29/10/2020 09:00:00', '4.4'],
['30/10/2020 09:00:00', '7.2'],
['31/10/2020 09:00:00', '8.8'],
['01/11/2020 09:00:00', '3.8'],
['02/11/2020 09:00:00', '3.4'],
['03/11/2020 09:00:00', '0.2'],
['04/11/2020 09:00:00', '0.0'],
['05/11/2020 09:00:00', '0.0'],
['06/11/2020 09:00:00', '0.2'],
['07/11/2020 09:00:00', '0.0'],
['08/11/2020 09:00:00', '9.8'],
['09/11/2020 09:00:00', '0.0'],
['10/11/2020 09:00:00', '2.8'],
['11/11/2020 09:00:00', '10.6'],
['12/11/2020 09:00:00', '1.8'],
['13/11/2020 09:00:00', '0.2'],
['14/11/2020 09:00:00', '3.4'],
['15/11/2020 09:00:00', '2.4'],
['16/11/2020 09:00:00', '0.8'],
['17/11/2020 09:00:00', '0.0'],
['18/11/2020 09:00:00', '0.8'],
['19/11/2020 09:00:00', '0.0'],
['20/11/2020 09:00:00', '1.8'],
['21/11/2020 09:00:00', '0.0'],
['22/11/2020 09:00:00', '0.0'],
['23/11/2020 09:00:00', '0.6'],
['24/11/2020 09:00:00', '1.2'],
['25/11/2020 09:00:00', '0.2'],
['26/11/2020 09:00:00', '0.0'],
['27/11/2020 09:00:00', '0.2'],
['28/11/2020 09:00:00', '0.0'],
['29/11/2020 09:00:00', '1.4'],
['30/11/2020 09:00:00', '1.6'],
['01/12/2020 09:00:00', '0.0'],
['02/12/2020 09:00:00', '1.8'],
['03/12/2020 09:00:00', '22.0'],
['04/12/2020 09:00:00', '19.8'],
['05/12/2020 09:00:00', '0.8'],
['06/12/2020 09:00:00', '0.4'],
['07/12/2020 09:00:00', '10.4'],
['08/12/2020 09:00:00', '2.8'],
['09/12/2020 09:00:00', '0.0'],
['10/12/2020 09:00:00', '7.4'],
['11/12/2020 09:00:00', '1.6'],
['12/12/2020 09:00:00', '1.2'],
['13/12/2020 09:00:00', '10.0'],
['14/12/2020 09:00:00', '0.0'],
['15/12/2020 09:00:00', '3.8'],
['16/12/2020 09:00:00', '1.0'],
['17/12/2020 09:00:00', '0.6'],
['18/12/2020 09:00:00', '5.4'],
['19/12/2020 09:00:00', '3.0'],
['20/12/2020 09:00:00', '0.0'],
['21/12/2020 09:00:00', '0.8'],
['22/12/2020 09:00:00', '0.0'],
['23/12/2020 09:00:00', '0.2'],
['24/12/2020 09:00:00', '0.0'],
['25/12/2020 09:00:00', '0.0'],
['26/12/2020 09:00:00', '4.2'],
['27/12/2020 09:00:00', '0.0'],
['28/12/2020 09:00:00', '2.6'],
['29/12/2020 09:00:00', '0.0'],
['30/12/2020 09:00:00', '0.0'],
['31/12/2020 09:00:00', '3.0'],
['01/01/2021 09:00:00', '0.0'],
['02/01/2021 09:00:00', '0.0'],
['03/01/2021 09:00:00', '6.6'],
['04/01/2021 09:00:00', '0.6'],
['05/01/2021 09:00:00', '0.2'],
['06/01/2021 09:00:00', '2.0'],
['07/01/2021 09:00:00', '0.6'],
['08/01/2021 09:00:00', '0.0'],
['09/01/2021 09:00:00', '1.8'],
['10/01/2021 09:00:00', '0.0'],
['11/01/2021 09:00:00', '9.6'],
['12/01/2021 09:00:00', '0.0'],
['13/01/2021 09:00:00', '15.4'],
['14/01/2021 09:00:00', '3.2'],
['15/01/2021 09:00:00', '9.2'],
['16/01/2021 09:00:00', '0.0'],
['17/01/2021 09:00:00', '0.6'],
['18/01/2021 09:00:00', '0.0'],
['19/01/2021 09:00:00', '9.4'],
['20/01/2021 09:00:00', '14.6'],
['21/01/2021 09:00:00', '1.0'],
['22/01/2021 09:00:00', '0.0'],
['23/01/2021 09:00:00', '0.0'],
['24/01/2021 09:00:00', '0.0'],
['25/01/2021 09:00:00', '0.6'],
['26/01/2021 09:00:00', '1.2'],
['27/01/2021 09:00:00', '0.8'],
['28/01/2021 09:00:00', '8.2'],
['29/01/2021 09:00:00', '6.4'],
['30/01/2021 09:00:00', '0.2'],
['31/01/2021 09:00:00', '0.0'],
['01/02/2021 09:00:00', '6.4'],
['02/02/2021 09:00:00', '11.0'],
['03/02/2021 09:00:00', '9.2'],
['04/02/2021 09:00:00', '29.4'],
['05/02/2021 09:00:00', '17.0'],
['06/02/2021 09:00:00', '4.4'],
['07/02/2021 09:00:00', '0.6'],
['08/02/2021 09:00:00', '0.8'],
['09/02/2021 09:00:00', '4.8'],
['10/02/2021 09:00:00', '5.0'],
['11/02/2021 09:00:00', '1.8'],
['12/02/2021 09:00:00', '0.8'],
['13/02/2021 09:00:00', '2.0'],
['14/02/2021 09:00:00', '2.8'],
['15/02/2021 09:00:00', '0.8'],
['16/02/2021 09:00:00', '2.4'],
['17/02/2021 09:00:00', '0.6'],
['18/02/2021 09:00:00', '5.8'],
['19/02/2021 09:00:00', '7.4'],
['20/02/2021 09:00:00', '1.4'],
['21/02/2021 09:00:00', '0.0'],
['22/02/2021 09:00:00', '0.2'],
['23/02/2021 09:00:00', '8.0'],
['24/02/2021 09:00:00', '0.2'],
['25/02/2021 09:00:00', '0.0'],
['26/02/2021 09:00:00', '0.0'],
['27/02/2021 09:00:00', '0.0'],
['28/02/2021 09:00:00', '0.0'],
['01/03/2021 09:00:00', '0.0'],
['02/03/2021 09:00:00', '0.0'],
['03/03/2021 09:00:00', '1.4'],
['04/03/2021 09:00:00', '0.2'],
['05/03/2021 09:00:00', '0.0'],
['06/03/2021 09:00:00', '0.2'],
['07/03/2021 09:00:00', '2.0'],
['08/03/2021 09:00:00', '0.0'],
['09/03/2021 09:00:00', '14.2'],
['10/03/2021 09:00:00', '4.6'],
['11/03/2021 09:00:00', '0.4'],
['12/03/2021 09:00:00', '1.2'],
['13/03/2021 09:00:00', '0.0'],
['14/03/2021 09:00:00', '0.0'],
['15/03/2021 09:00:00', '2.2'],
['16/03/2021 09:00:00', '0.0'],
['17/03/2021 09:00:00', '0.0'],
['18/03/2021 09:00:00', '0.0'],
['19/03/2021 09:00:00', '0.2'],
['20/03/2021 09:00:00', '0.0'],
['21/03/2021 09:00:00', '0.0'],
['22/03/2021 09:00:00', '0.0'],
['23/03/2021 09:00:00', '0.0'],
['24/03/2021 09:00:00', '2.4'],
['25/03/2021 09:00:00', '3.4'],
['26/03/2021 09:00:00', '0.6'],
['27/03/2021 09:00:00', '3.0'],
['28/03/2021 09:00:00', '1.6'],
['29/03/2021 09:00:00', '0.0'],
['30/03/2021 09:00:00', '0.0'],
['31/03/2021 09:00:00', '0.0'],
['01/04/2021 09:00:00', '0.0'],
['02/04/2021 09:00:00', '0.0'],
['03/04/2021 09:00:00', '0.0'],
['04/04/2021 09:00:00', '0.4'],
['05/04/2021 09:00:00', '0.2'],
['06/04/2021 09:00:00', '0.0'],
['07/04/2021 09:00:00', '0.0'],
['08/04/2021 09:00:00', '0.2'],
['09/04/2021 09:00:00', '0.0'],
['10/04/2021 09:00:00', '1.2'],
['11/04/2021 09:00:00', '0.2'],
['12/04/2021 09:00:00', '0.0'],
['13/04/2021 09:00:00', '0.0'],
['14/04/2021 09:00:00', '0.2'],
['15/04/2021 09:00:00', '0.0'],
['16/04/2021 09:00:00', '0.0'],
['17/04/2021 09:00:00', '0.0'],
['18/04/2021 09:00:00', '0.0'],
['19/04/2021 09:00:00', '0.0'],
['20/04/2021 09:00:00', '0.4'],
['21/04/2021 09:00:00', '0.0'],
['22/04/2021 09:00:00', '0.0'],
['23/04/2021 09:00:00', '0.0'],
['24/04/2021 09:00:00', '0.0'],
['25/04/2021 09:00:00', '0.0'],
['26/04/2021 09:00:00', '3.6'],
['27/04/2021 09:00:00', '1.6'],
['28/04/2021 09:00:00', '0.2'],
['29/04/2021 09:00:00', '0.6'],
['30/04/2021 09:00:00', '0.0'],
['01/05/2021 09:00:00', '3.0'],
['02/05/2021 09:00:00', '0.2'],
['03/05/2021 09:00:00', '30.0'],
['04/05/2021 09:00:00', '1.0'],
['05/05/2021 09:00:00', '0.6'],
['06/05/2021 09:00:00', '5.6'],
['07/05/2021 09:00:00', '1.4'],
['08/05/2021 09:00:00', '6.8'],
['09/05/2021 09:00:00', '0.6'],
['10/05/2021 09:00:00', '4.8'],
['11/05/2021 09:00:00', '9.4'],
['12/05/2021 09:00:00', '0.2'],
['13/05/2021 09:00:00', '0.0'],
['14/05/2021 09:00:00', '0.0'],
['15/05/2021 09:00:00', '0.0'],
['16/05/2021 09:00:00', '0.0'],
['17/05/2021 09:00:00', '8.0'],
['18/05/2021 09:00:00', '1.0'],
['19/05/2021 09:00:00', '10.0'],
['20/05/2021 09:00:00', '18.2'],
['21/05/2021 09:00:00', '3.4'],
['22/05/2021 09:00:00', '0.2'],
['23/05/2021 09:00:00', '32.8'],
['24/05/2021 09:00:00', '3.2'],
['25/05/2021 09:00:00', '2.4'],
['26/05/2021 09:00:00', '1.8'],
['27/05/2021 09:00:00', '0.0'],
['28/05/2021 09:00:00', '0.2'],
['29/05/2021 09:00:00', '0.0'],
['30/05/2021 09:00:00', '0.0'],
['31/05/2021 09:00:00', '0.0'],
['01/06/2021 09:00:00', '0.0'],
['02/06/2021 09:00:00', '0.0'],
['03/06/2021 09:00:00', '0.0'],
['04/06/2021 09:00:00', '0.0'],
['05/06/2021 09:00:00', '0.0'],
['06/06/2021 09:00:00', '0.0'],
['07/06/2021 09:00:00', '0.0'],
['08/06/2021 09:00:00', '0.0'],
['09/06/2021 09:00:00', '0.0'],
['10/06/2021 09:00:00', '0.2'],
['11/06/2021 09:00:00', '0.0'],
['12/06/2021 09:00:00', '0.0'],
['13/06/2021 09:00:00', '3.2'],
['14/06/2021 09:00:00', '0.0'],
['15/06/2021 09:00:00', '0.0'],
['16/06/2021 09:00:00', '0.0'],
['17/06/2021 09:00:00', '0.0'],
['18/06/2021 09:00:00', '0.0'],
['19/06/2021 09:00:00', '0.0'],
['20/06/2021 09:00:00', '0.0'],
['21/06/2021 09:00:00', '0.2'],
['22/06/2021 09:00:00', '0.0'],
['23/06/2021 09:00:00', '0.0'],
['24/06/2021 09:00:00', '3.4'],
['25/06/2021 09:00:00', '2.4'],
['26/06/2021 09:00:00', '7.4'],
['27/06/2021 09:00:00', '1.0'],
['28/06/2021 09:00:00', '0.0'],
['29/06/2021 09:00:00', '0.0'],
['30/06/2021 09:00:00', '0.0'],
['01/07/2021 09:00:00', '0.0'],
['02/07/2021 09:00:00', '0.0'],
['03/07/2021 09:00:00', '1.0'],
['04/07/2021 09:00:00', '5.8'],
['05/07/2021 09:00:00', '1.0'],
['06/07/2021 09:00:00', '5.6'],
['07/07/2021 09:00:00', '0.2'],
['08/07/2021 09:00:00', '0.0'],
['09/07/2021 09:00:00', '0.0'],
['10/07/2021 09:00:00', '0.4'],
['11/07/2021 09:00:00', '0.2'],
['12/07/2021 09:00:00', '0.0'],
['13/07/2021 09:00:00', '0.0'],
['14/07/2021 09:00:00', '0.0'],
['15/07/2021 09:00:00', '0.0'],
['16/07/2021 09:00:00', '0.0'],
['17/07/2021 09:00:00', '0.0'],
['18/07/2021 09:00:00', '0.0'],
['19/07/2021 09:00:00', '0.0'],
['20/07/2021 09:00:00', '0.0'],
['21/07/2021 09:00:00', '0.2'],
['22/07/2021 09:00:00', '0.0'],
['23/07/2021 09:00:00', '0.0'],
['24/07/2021 09:00:00', '0.0'],
['25/07/2021 09:00:00', '0.0'],
['26/07/2021 09:00:00', '0.0'],
['27/07/2021 09:00:00', '12.6'],
['28/07/2021 09:00:00', '0.6'],
['29/07/2021 09:00:00', '1.0'],
['30/07/2021 09:00:00', '4.8'],
['31/07/2021 09:00:00', '0.8'],
['01/08/2021 09:00:00', '0.0'],
['02/08/2021 09:00:00', '0.0'],
['03/08/2021 09:00:00', '0.0'],
['04/08/2021 09:00:00', '0.0'],
['05/08/2021 09:00:00', '11.0'],
['06/08/2021 09:00:00', '1.2'],
['07/08/2021 09:00:00', '1.6'],
['08/08/2021 09:00:00', '5.4'],
['09/08/2021 09:00:00', '1.8'],
['10/08/2021 09:00:00', '13.2'],
['11/08/2021 09:00:00', '4.8'],
['12/08/2021 09:00:00', '0.2'],
['13/08/2021 09:00:00', '0.0'],
['14/08/2021 09:00:00', '0.0'],
['15/08/2021 09:00:00', '0.0'],
['16/08/2021 09:00:00', '0.2'],
['17/08/2021 09:00:00', '0.0'],
['18/08/2021 09:00:00', '2.8'],
['19/08/2021 09:00:00', '0.2'],
['20/08/2021 09:00:00', '3.2'],
['21/08/2021 09:00:00', '8.0'],
['22/08/2021 09:00:00', '0.0'],
['23/08/2021 09:00:00', '0.0']]
Note that this has read in all of the data as strings, and has included the headers. We could get rid of the headers by slicing the list (rain_data = rain_data[1:]), and we could convert the values to floats by using float. Here’s how we might do both simultaneously using list comprehension:
rain_data = [[x[0], float(x[1])] for x in rain_data[1:]]
Note that this has left the timestamp as a string. Python does have the ability to work directly with so called datetimes, but we leave this out for simplicity.
Using pandas#
If you are planning to do statistical analysis on your data, you are likely to want to import it into a “dataframe” using pandas instead:
import pandas as pd
rain_data2 = pd.read_csv("data/strathkiness-rainfall.csv")
rain_data2
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 import pandas as pd
3 rain_data2 = pd.read_csv("data/strathkiness-rainfall.csv")
4 rain_data2
ModuleNotFoundError: No module named 'pandas'
Note that this is stored in a dataframe object, rather than a list. You could extract the values out of the dataframe object like so:
rain_data2.values
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 rain_data2.values
NameError: name 'rain_data2' is not defined
Note that pandas has automatically converted the second column to float. This makes pandas a convenient way of reading in csv files even if you don’t plan to use any other pandas features.
After manipulating the data using pandas, you might want to write the dataframe back to a csv file.
# rain_data2.to_csv("data/processed-rainfall-data.csv")
Numpy input and output#
Often the data we want to read or write to a file comes in the form of a numpy array. For example, after computing the values of a function on a large and high-resolution mesh, you might want to avoid recomputing them. The example below doesn’t take long to compute, but it will take a lot longer if the number of points in the linspace increases significantly.
import numpy as np
X = Y = np.linspace(0, 10, 500)
X, Y = np.meshgrid(X, Y)
Z = 3 * X**2 * Y**3 + 27 * X
# now that we've computed these values, we can save them:
np.savetxt("data/Z.txt", Z)
# If there is a lot of data, you might want to save a compressed file instead.
# Just add ".gz" to the end of the file name
# np.savetxt("data/Z.txt.gz", Z)
Now that we’ve saved the files, we can easily load them in at any later point:
X = Y = np.linspace(0, 10, 500)
X, Y = np.meshgrid(X, Y)
Z = np.loadtxt("data/Z.txt")