Data Flow

I have spent a long time at the analysis stage of data but I really wanted to understand how data flows from source to the final report. I found this website had a great visual that helped me understand the different steps. Once I had a general idea of the flow I explored all details separately. Here is my understanding of how data goes from the source to using it for either visualizing analysis or doing AI/ML models.

In order to understand all the terms, such as OLTP vs OLAP, data lake vs warehouse, ELT vs ETL, I wrote a detailed comparison on my versus page!

The Lambda Function

It is also sometimes referred to as an “Anonymous Function”, or as I like to call it the “Throwaway Function”. This allows you to create a function within the code as you normally might, but with less rigour. So technically this is a function, but an odd-instance of a function as you will not define it using “def” but rather “lambda” keyword. The reason I call it a “Throwaway Function” is because you really only use it if you want to do it once, unlike other functions, which are created for the purpose of reusability.

• Example

  The first simple example can be if you wanted to multiply a number by 2

  lambda x: x*2

  You have now defined an anonymous function. If we want it to directly work on a number we would just have to put the number we want after:

  (lambda x: x*2) (4)

  Our output would be 8.
  
  Now we can use a similar logic to a list:

  number_list = [2, 5, 7, 1]

  list(map(lambda number: number*2, number_list))

  Our output will be [4, 10, 14, 2]
  
  I have found lambda functions come in handy when you start working with datasets where you just need to change one particular aspect of one set of data points.
  
  More details on the function can be found here and useful built-in functions here

IPython

IPython is a shell for interactive computing. This can be done directly in the terminal. It lets you see the line by line interaction of your code in real time.

There are such programs, most popular being Jupyter, that are also used partly for their interactive computing.

Bypassing Cookies

Cookies are found on websites that store data in order to allow you to access specific parts of certain websites. E.g. Costco.ca has cookies for many things, one thing in particular is your location. By saving your location in the cookies it allows Costco to show deals for your specific province/territory. I started looking into cookies in order to help me with my Costco app.

• Methods

  Work around for them (I followed this example):
  In order to list all our cookies from a URL, we can:

  page.cookies

  We make a for loop in order to cycle through the information for the cookies we need

  for cookie in page.cookies :
  print('cookie domain: ' + cookie.domain)
  print('cookie name: ' + cookie.name)
  print('cookie value: ' + cookie.value)
  print('----------------')

  To input cookies to a url:

  Method #1:
  First, create a dictionary object

  cookies = dict(name ='anja', password = 'pass')

  Then send cookies to the url with the cookie parameters we want

  page = requests.get(URL, cookies = cookies)

  
  

  Method #2:
  In order to be able to populate and set multiple cookies on a webpage, we must create a RequestsCookieJar object

  cookies_jar = requests.cookies.RequestsCookieJar()

  Then we add each cookie to the cookies_jar

  cookies_jar.set (‘name’, ‘anja’, domain = ‘URL’, path = ‘/cookies’)
  cookies_jar.set (‘password’, ‘pass’, domain = ‘URL’, path = ‘/cookies’)

  Finally, we send the cookies to the url with the cookie parameters we want

  Response = requests.get(URL, cookies = cookies_jar)

  This ended up not working for my Costco deal finder, however, I did learn a lot! I had to use the Selenium package in order to input the data of the location to proceed with the app.

How to Solve Errors

Along the way I had been given errors and I had a hard time figuring out where they came from. I found that you can use “try”, “except” and “breakpoint()” to try to pinpoint the errors.
→ breakpoint() lets you step through the function line-by-line to figure out where your mistake might be.
→ try is to let the program go until you reach a problem. This saves time instead of doing breakpoint because you don't need to step through your whole program, just the broken part. In order to do this, you must try a line of code and then tell it what to do in the except section.

Markdown Files

.md files stand for markdown. They are basically fancy text files that let you add formatting/style to them. You can make them as their own separate files, or GitHub has integrated them in their comments for issues or on pull requests. This makes it really useful to use if you use GitHub collaboratively.

There are a bunch of ways you can format them and GitHub made a list. There is also a great "cheat sheet" made by Square Space or the Markdown Guide.