Submitted by harishprab t3_yga0s1 in MachineLearning
We have recently open sourced our inference acceleration library, voltaML.
⚡VoltaML is a lightweight library to convert and run your ML/DL deep learning models in high performance inference runtimes like TensorRT, TorchScript, ONNX and TVM.
We would love for the reddit and the open-source community to use it, give feedback and help us improve the library.
Oh I guess the library supports accelerating a couple of well known models defined in the models module?
Looks promising. A comparison with other competitors (hf accelerate, neuralmagic, nebullvm, ...) would be great
Do you have any benchmarks against other frameworks and have you benchmarked other types of models or are you doing something specific for NLP?
Hi, I am one of the authors of transformer deploy. I have seen you have copied most of the files for the transformer part. That’s really cool, I really appreciate you kept the licenses, may I ask you to cite our work in the Readme ?
Moreover, if I may, why did you copied instead of just importing a dependency? You would get the maintenance for free :-)
Hey. You’ve done amazing work with transformers deploy. We have actually mentioned you in the work. We just wanted to have voltaML to be one repo for all ML, CV and NLP needs.
We use the TorchFX library to do this on CPU. And TensorRT is doing this for GPU. We’re not using any custom function for the fusing. TorchFX and TensorRT are doing it anyways
Thanks. HF accelerate is basically doing it for intel chips. I haven’t seen them support TensorRT. I could be wrong. Neural Magic is mostly about quantisation aware training and pruning techniques. We focus on post training techniques. We should try Nebullvm. They’re a great library too.
We have computer vision, NLP and decision trees inference acceleration.
Maybe we’ll have it as a dependency. We’re also planning to do some of our own work on NLP. So thought we’ll have it as a non dependency
Oh wow, I have no idea how I missed the other parts of the readme that shows other types of applications. Do you plan on showing any benchmarks against other frameworks?
I understand that. But let’s say I have these operators named as c1, b1, r1 instead of what it expects, the fusion logic won’t work. So my question was if this library works only a fixed set of models defined in the library itself or it can work against any models users write.
Right now it supports only for the models that are supported by these libraries. We have tried fusion manually earlier but ran into many issues given the diversity of models. So we stuck to torchfx and trt. Maybe in the future we can make it modular so that it can work on any model.
PlayOffQuinnCook t1_iu7ualp wrote
Congrats on open sourcing! Quick question on fusion - how do you guys fuse layers like conv, bn, relu, etc, if they are not named conv1, bn1, relu in the nn.Module.