[D] Pytorch or TensorFlow for development and deployment?

Submitted by CodaholicCorgi t3_z50vvq in MachineLearning

Hi folks,

I know this question is kept being asked for many times, but as time change, and Pytorch is getting more favorable everywhere, I want to bring this one on the table again.

Assuming that you are building AI products (deployment is a must!), do you prefer TensorFlow or Pytorch in your codebase and why?

For me (and most likely a lot of people out there), I prefer training and developing my models in Pytorch (ease of debugging and customizing). But if that's a case, is there any option to deploy the .pt models? Even if we need .tflite for mobile deployment?

Thanks in advance!

66

Comments

You must log in or register to comment.

Deep-Station-1746 t1_ixu2bxb wrote

First, PlayTorch is a thing. That could replace tflite for mobile.

Second, you can use ONNX to transfer models from pt to tf, then to tflite.

Last but not least, if you are a REAL MAN, you can try to translate the model weights to tf by hand, using numpy as an intermediary.

71

Deep-Station-1746 t1_ixu6ych wrote

Replying to Background_Thanks604 (#682,622)

Load pt model with torch, get all weights with state_dict. Make them all into channels last format with permute. Load all the arrays into numpy. Load all the arrays into tensorflow. Write the network forward logic in tensorflow. Plug in weights. Run.

31

sarmientoj24 t1_ixukgwd wrote

Not a big TF guy but thr SOTA models nowadays will usually be in Pytorch. So if you need those, then Pytorch.

38

a1_jakesauce_ t1_ixvn6jf wrote

Replying to sarmientoj24 (#684,942)

I don’t know much about this topic but to me, it seems there is a difference between a SOTA model developed for a paper and a model ready for deployment. Seems torch is easier to dev while tf uses lazy evaluation and is therefore lighter. Could be wrong

2

a1_jakesauce_ t1_ixvnw93 wrote

Replying to kaskoosek (#690,756)

I don’t know much about this either but it seems you’re right since there’s not a one size fits all solution for anything. But at the same time, it’s not very helpful advice

1

kaskoosek t1_ixvor9u wrote

Replying to a1_jakesauce_ (#690,808)

Definitely not helpful advise. Anyway it feels like the question caters to fanboys of each enviroment.

Both pytorch and tensorflow are pretty good, though each has their own strengths.

1

sarmientoj24 t1_ixvouw5 wrote

Replying to a1_jakesauce_ (#690,695)

The thing is, either you wait for the TF support or you code it yourself. Research is fast paced so researchers want a boilerplate, or just to build on top of another repo.

For deployment, there's a lot of deployment tools for Pytorch nowadays. I use BentoML for deploying compvision models. If you want a lighter model, there are libraries and repositories that support sparsification, pruning, etc. of your models. A lot of newer repos for industrial use (mmdet, YOLOv5/6/7) have sparsification and pruning support. Some of them even have distillation built-in. Again, I am not a TF guy so I havent seen such rich support for model deployment than these Pytorch-based repos.

While it is true that research and deployment are different, it is up to your team how to do MLOps on those SOTA methods. New SSL method, new loss functions, new stuff -- either you borrow it from a repo or you code it yourself. In the industry, the prototyping procedure for experimentatio is critical for a fast-moving product. You dont have time to convert codes. So you test repositories, then build on top of it if it is successful.

6

erannare t1_ixvt17s wrote

Replying to sarmientoj24 (#690,918)

TensorFlow has model optimization libraries such as, but not limited to: weight clustering, pruning, and weight quantization, as well as training support for these.

2

sarmientoj24 t1_ixw1ca2 wrote

Replying to erannare (#691,401)

That's great. Again, I am not used to TF. My point was that Pytorch has deployment capabilities near TF's nowadays and that most SOTA and new research are leaning towards publishing their code with Pytorch.

1

wisscool t1_ixw4fd7 wrote

Develop in whatever framework you like, then deploy with Triton inference server

−1

ApeForHire t1_ixwveuz wrote

Replying to Background_Thanks604 (#683,726)

I was actually able to do this once by relying heavily on Github's Copilot, ie just giving functions very specific names and commenting things like "this function converts a pytorch model into a vector of weights" etc. It worked pretty well and was simple for basic network architectures, though I imagine it could get more complicated.

1

nwestninja t1_ixx5ewa wrote

I usually do onnx or tensorRT for deployment and use whatever I want for deployment. Most libraries can serialize down to an ONNX model or a tensorRT model.

1

rsandler t1_ixxu6px wrote

My 2 cents: I am still the stubborn minority using TF. Mostly because of switching costs in my organization. But also, honestly, because I am decently happy with it. I think the ease-of-use gap between pytorch and TF was mostly erased with the release of TF2. I was really hoping that more of the research community would come back to TF after that, but unfortunately that has not been the case. In practice, it does suck to not have access to SOTA models. Otherwise, I think they are pretty similar.

3

CodaholicCorgi OP t1_iy1s4ws wrote

Replying to Background_Thanks604 (#682,622)

I did it once in one of my projects. It's basically hand-picking weights from Pytorch model's layers to TensorFlow model's layers, but it feels more reliable than relying on ONNX with a bunch of warnings.

Though there are not much tutorials or blog posts about this, I will try creating a github repo for this later (just examples with simple layers), so many more people will know that this technique exists.

2