Midwest Dev Chat

MidwestDevChat.com github.com/skidvis/Midwest-Dev-Chat

  • The newest 15 messages in the super-cool #database channel.

  • 01/18 07:12:56 Laverna: Not every time, because mongo will cache it. Depending on your app architecture you may be able to do that as well.

  • 01/18 07:13:38 Laverna: Now, the REAL question is: do you really need all 250mb every time you access?

  • 01/18 07:14:20 Amalia: once i load the variable once, i dont need it anymore

  • 01/18 07:15:31 Laverna: Do you are already caching in-app. Disk will be fine.

  • 01/18 07:16:31 Laverna: 20s seems very long though. Either you are measuring badly, your code is doing too much, or you have a bad/very old disk

  • 01/18 07:17:19 Amalia: i see

  • 01/18 07:17:24 Amalia: let me check again

  • 01/18 07:23:00 Amalia: 18 seconds using `np.genfromtxt()`

  • 01/18 07:35:13 Amalia: uploaded a file: https://midwestdevchat.slack.com/files/lordlabakdas/F3TPAUMJA/-.py

  • 01/18 08:40:53 Laverna: Is that in seconds or milliseconds?

  • 01/18 09:09:24 Laverna: Also, if you are really concerned about benchmarking this, you might want to split out the operations of loading the file and processing it with numpy. Then you will see the true(r) costs.

  • 01/18 10:01:11 Lauretta: @Amalia if you're just trying to put this into memory, why not use a ramdisk?

  • 01/18 10:12:04 Laverna: I was avoiding the optimizations until we had established a baseline wink

  • 01/18 10:17:15 Amalia: a few optimizations on the code side (http://scikit-learn.org/stable/modules/model_persistence.html) helped solve my issue.....i could save the state of my model offline

  • 01/18 10:23:32 Laverna: I wondered about that. Glad to hear it @Amalia

  • *Usernames have been changed to protect the innocent.

Check out all the cool channels!

Join the conversation!