Submitted by immune_star t3_11yh8x8 in MachineLearning
Released the data and code used to train CodeAlpaca - https://github.com/sahil280114/codealpaca
Comments
immune_star OP t1_jda1ri4 wrote
Good point, i'll open source the weights tomorrow as well
2muchnet42day t1_jda64te wrote
!RemindMe 1 day
RemindMeBot t1_jda68wu wrote
I will be messaging you in 1 day on 2023-03-23 22:49:36 UTC to remind you of this link
3 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) ^(delete this message to hide from others.)
| ^(Info) | ^(Custom) | ^(Your Reminders) | ^(Feedback) |
|---|
RemarkableGuidance44 t1_jdc2opy wrote
Yeah I was wondering why you did not release them as its allowed as you are not selling it. :)
2muchnet42day t1_jdgy2ht wrote
Are they public now?
2muchnet42day t1_jd8u97a wrote
We need to start iterating on the same weights and not start from scratch everytime
2muchnet42day t1_jd7upsm wrote
It's awesome. Thank you for your work.
I'd like to know why you didn't take the LoRA approach to finetuning LLaMA? Is a full finetuning better?
immune_star OP t1_jd892n1 wrote
Primarily I had the hardware needed to do a full finetune so just went ahead with it, also LoRA can lead to a slight loss in quality.
2muchnet42day t1_jd8fnje wrote
Would you consider doing a LoRA version of CodeAlpaca and compare the ouputs of the two models?
RemarkableGuidance44 t1_jdc3hut wrote
Yeah I would to know what the difference is from LoRA to just Full finetune?
radi-cho t1_jd8gdzt wrote
Great. Congratulations. I was planning on attempting the same basically, so thanks for open-sourcing it:)
StablePunFusion t1_jd8qm6b wrote
Thanks for releasing the training data (https://github.com/sahil280114/codealpaca/blob/master/data/code_alpaca_20k.json).
Where was the training data gathered from? Has the data been verified to be correct?
I'm a tad sad to see that most of the training data doesn't have the language tagged anywhere, some do but most don't, so the resulting model might not be super useful as it'll confuse languages, I guess.
immune_star OP t1_jd8qqsg wrote
Data has been generated using text-davinci-003 , not verified to be correct
StablePunFusion t1_jd8r3xb wrote
Do you (or anyone) know of any higher quality sources of training sets for code?
Seems to be lacking, at least when I searched around last time. Maybe it's time to spin up a community initiative around it?
[deleted] t1_jd8tbce wrote
[removed]
Nice_Cod7781 t1_jd8avf1 wrote
Why release without the weights? All it does is force people to expend extra energy and time on something that could have been provided originally. It's bad from a cooperative perspective and doesn't help the environment either.
You're not commercializing this so it's not like you're going to get into any legal trouble for releasing the model.