Chapter 2 - Questionnaire

Vishal Bakshi
Friday, August 28, 2020

I will start Chapter 2 by filling out the questionnaire, before reading the text or watching the lesson video.

Q1. Provide an example of where the bear classification model might work poorly in production, due to structural or style differences in the training data.

  • One example of this, I think, is that if the style of bear image (cartoon, painting, etc) is different than what is seen in production.

Q2. Where do text models currently have a major deficiency?

  • I would guess: unstructured text

Q3. What are possible negative societal implications of text generation models?

  • At the end of the day, models trained by data generated by humans will have the same flaws as us, but potentially (and eventually) more power to push those flaws farther without self-correction. This includes our biases, prejudices, ignorance, hatred, self-interest and so on.

Q4. In situations where a model might make mistakes, and those mistakes could be harmful, what is a good alternative to automating a process?

  • Adding a step of human validation before taking an action or making a decision based on the prediction

Q5. What kind of tabular data is deep learning particularly good at?

  • Not sure about this. I read about collaborative filtering in this article by Jeremy Howard, Margit Zwemer, and Mike Loukides. I would guess that one requirement for tabular data is that it should be structured in a way where you can perform collaborative filtering ? Maybe that's only a requirement for recommendation systems.
    • I would also guess that deep learning is better at predicting when some of the inputs are categorical?

Q6. What's a key downside of directly using a deep learning model for recommendation systems?

  • My guess would be that the deep learning model will make predictions not recommendations, an optimizer is needed to supplement the prediction. From the Drivetrain approach article: "Prediction only tells us that there is going to be an accident. An optimizer tells us how to avoid accidents." So we would need to pass the prediction through an optimizer to avoid accidents (predictions that moves us away from the objective).

Q7. What are the steps of the Drivetrain Approach?

  • Defined objective > Levers (what inputs can we control) > Data (what data we can collect) > Models (how data influences the objective)
  • Step 4 (Models) is further broken down into the following:
    • Objective > Data > Modeler > Simulator > Optimizer > Actionable Outcome

Q8. How do the steps of the Drivetrain Approach map to a recommendation system?

  • Objective (drive sales from customer) > Levers (ranking of recommendations) > Data (randomized experiments on different recommendation effects on sales for different customers) > Model (conditional on seeing or not seeing a recommendation) > Optimizer (ranks books based on simulated utility to meet objective)

Q9. Create an image recognition model using data you curate, and deploy it on the web.

Q10. What is DataLoaders?

  • An object that wraps two DataLoader objects: one with training data and one with validation data.
In [1]:
from fastai.vision.all import *

Q11. What four things do we need to tell fastai to create DataLoaders?

  • The path to the data, the filenames fnames, how to get labels using label_func and the batch size bs
In [4]:
doc(ImageDataLoaders.from_name_func)
In [5]:
doc(DataBlock)

Q12. What does the splitter parameter to DataBlock do?

  • splitter seems to be a function that takes the full dataset and splits it into subsets, which are then wrapped into a Datasets object and returned on a DataLoaders.datasets call.

Q13. How do we ensure a random split always gives the same validation set?

  • use the same number for the random seed each time
  • np.random.seed(int)

Q14. What letters are often used to signify the independent and dependent variables?

  • x (independent)
  • y (dependent)

Q15. What's the difference between the crop, pad, and squish resize approaches? When might you choose one over the others?

  • this blog post
  • vision.augment docs
    • crop removes pixels until the image fits whatever size is chosen for the training
    • pad shrinks the image until the longest dimension fits inside the chosen size, then fills the gap between the original image and the area of size with a reflection of the image (by default) with other options (like padding with zeroes which keeps a black border around the image).
    • squish will distort the image so that it fits within size which will change the proportions of the image

Q16. What is data augmentation? Why is it needed?

  • Data augmentation is the process of adding data to the dataset by creating transformed copies of the original data. This can take the form of flipping, rotating, warping, zooming or resizing the image.

Q17. What is the difference between item_tfms and batch_tfms?

  • item_tfms: a transform applied to a single image
  • batch_tfms: a transform applied to a batch

Q18. What is a confusion matrix?

  • A nxn matrix (where n is the number of outputs or categories) comparing how many times the predicted label for an image matched either the actual label of the correct image or the actual label of an incorrect image. The confusion matrix for a model predicting dogs and cats would show the number of times actual dogs were predicted as dogs, actual dogs were predicted as cats, actual cats were predicted as cats, and actual cats were predicted as dogs.

Q19. What does export save?

  • By default, export saves a .pkl file of the model

Q20. What is it called when we use a model for getting predictions, instead of training?

  • Not sure about this one

Q21. What are IPython widgets?

  • GUIs you can implement to a Jupyter Notebook to add functionality.
  • from the docs: "Widgets are eventful python objects that have a representation in the browser, often as a control like a slider, textbox, etc."

Q22. When might you want to use CPU for deployment? When might GPU be better?

  • a CPU would be better for processing single requests quickly, whereas a GPU would be better for processing many requests at the same time

Q23. What are the downsides of deploying your app to a server, instead of to a client (or edge) device such as a phone or PC?

  • i think that one downside is cost---your server has to be running in order to receive and process requests which can add up cost over time.
  • another downside is time---the model has to be loaded and there is an additional time of uploading the new data in the web app.
  • a third downside is you have to have a reliable internet connection to use the model.

Q24. What are three examples of problems that could occur when rolling out a bear warning system in practice?

  • not sure about this:
    • non-bears are identified as bears, triggering a false positive warning
    • bears are identified as non-bears, not triggering the warning, resulting in a false negative
    • the bear image or video does not resemble what the model was trained on

Q25. What is "out-of-domain data"?

  • data that the model was not trained on

Q26. What is "domain shift"?

  • not sure, but probably what happens in transfer-learning, as the model's domain shifts?

Q27. What are the three steps in the deployment process?

  • not sure but in general:
    • host a VM somewhere to run an environment for your server
    • save your model somewhere where it can be requested
    • deploy your app
In [ ]: