中文版本

新模型推荐

序号	模型名称&快捷链接
0	Yi-34B-Chat-4bits
1	Yi-34B-Chat-8bits
2	Yi-6B-Chat
3	Yi-6B-Chat-4bits
4	Yi-6B-Chat-8bits
5	Yi-34B-Chat
6	Video-LLaVA-V1.5
7	Video-LLaVA-7B
8	LanguageBind_Video
9	LanguageBind_Video_FT
10	LanguageBind_Image
11	LanguageBind_Video_merge
12	LanguageBind_Audio
13	LanguageBind_Depth
14	LanguageBind_Thermal
15	Video-LLaVA-Pretrain-7B
16	Aquila2-70B-Expr
17	AquilaChat2-70B-Expr
18	AquilaChat2-34B-Int4-GPTQ
19	AquilaChat2-34B-16K
20	speech_rwkv_transducer_asr-en-16k-gigaspeech-vocab5001-pytorch-online
21	funasr-runtime-win-cpu-x64
22	ModelScope-Agent-14B
23	speech_sambert-hifigan_nsf_tts_donna_en-us_24k
24	speech_sambert-hifigan_nsf_tts_david_en-us_24k
25	MiniGPT-v2
26	speech_sambert-hifigan_tts_waan_Thai_16k
27	cv_background_generation_sd
28	speech_eres2net_base_250k_sv_zh-cn_16k-common
29	Pose-driven-image-generation-HumanSD
30	cv_stable-diffusion-v2_image-feature
31	nlp_minilm_ibkd_sentence-embedding_english-sts
32	nlp_minilm_ibkd_sentence-embedding_english-msmarco
33	speech_eres2net_large_mej_lre_16k_common
34	speech_eres2net_base_mej_lre_16k_common
35	Ziya2-13B-Base
36	SUS-Chat-34B
37	animatediff-motion-adapter-v1-5
38	animatediff-motion-adapter-v1-4
39	animatediff-motion-adapter-v1-5-2
40	animatediff-motion-lora-zoom-in
41	animatediff-motion-lora-pan-left
42	animatediff-motion-lora-tilt-up
43	animatediff-motion-lora-rolling-clockwise
44	animatediff-motion-lora-zoom-out
45	animatediff-motion-lora-pan-right
46	animatediff-motion-lora-tilt-down
47	animatediff-motion-lora-rolling-anticlockwise
48	Qwen-1_8B-Chat-Int4
49	Qwen-1_8B-Chat-Int8
50	Qwen-72B-Chat-Int4
51	Qwen-72B-Chat-Int8
52	Qwen-Audio-Chat
53	Qwen-72B-Chat
54	Qwen-72B
55	Qwen-1_8B
56	Qwen-1_8B-Chat
57	Qwen-Audio
58	cogvlm-chat
59	cogvlm-base-224
60	cogvlm-base-490
61	cogvlm-grounding-base
62	cogvlm-grounding-generalist
63	deepseek-llm-7b-base
64	deepseek-llm-67b-base
65	deepseek-llm-7b-chat
66	deepseek-llm-67b-chat
67	xlm-MLM-en-2048
68	OrionStar-Yi-34B-Chat
69	tigerbot-180b-base-v2
70	tigerbot-13b-chat-v5
71	tigerbot-13b-chat-v5-4bit-exl2
72	tigerbot-70b-chat-v4-4bit-exl2
73	tigerbot-180b-chat-v2
74	tigerbot-13b-chat-v5-4k
75	tigerbot-13b-base-v3
76	tigerbot-70b-chat-v4-4k
77	tigerbot-70b-base-v2
78	tigerbot-70b-chat-v4
79	BlueLM-7B-Base
80	BlueLM-7B-Chat-32K
81	BlueLM-7B-Chat-4bits
82	Sunsimiao-Qwen-7B
83	MindChat-Qwen-7B-v2-self_lora
84	jina-embeddings-v2-base-en
85	jina-embeddings-v2-small-en
86	qwen-chat-7B-ggml
87	qwen-chat-14B-ggml
88	bge-reranker-large
89	bge-reranker-base

高亮功能

• 支持本地拉起测试推理服务
• 支持vllm推理
• LLMPipeline 支持vllm
• 官方镜像升级到python3.10，pytorch升级2.1.0, tensorflow 1.14.0 ubuntu22.04
• upgrade to python3.10

功能列表

• Support VLLM in LLMPipeline (#604)
• add bpemodel path in asr_trainer
• add llm riddles (#621)
• feat: deploy checker for swingdeploy

功能提升

• python311 support for whl
• llm pipeline support chatglm3 (#618)
• Support transformers==4.35.0 (#633)

BugFix

• Fix _set_gradient_checkpointing bug (#660)
• fix test reliability issue (#657)
• fix: DocumentGroundedDialogRetrievalModel qry_encoder.encoder.embeddings.position_ids error (#647)
• fix asr paraformer finetune bug
• fix uie trainer: eval failed (#617)
• Fix vllm: change if condition (#607)
• fix shop_segmentation to use old timm lib and bump version to 1.9.4rc2
• fix the numpy bug for card detection correction
• fix issues for 3dhuman models
• fix logger: remove file handler for original user logging (#645)

English Version

Highlight

• local launch inference server
• support vllm
• LLMPipeline support vllm
• Image upgrade to python3.10, pytorch2.1.0,tensorflow2.14.0, ubuntu22.04

Breaking changes

Feature

• Support VLLM in LLMPipeline (#604)
• add bpemodel path in asr_trainer
• add llm riddles (#621)
• feat: deploy checker for swingdeploy

Improvements

• python311 support for whl
• llm pipeline support chatglm3 (#618)
• Support transformers==4.35.0 (#633)

BugFix

• Fix _set_gradient_checkpointing bug (#660)
• fix test reliability issue (#657)
• fix: DocumentGroundedDialogRetrievalModel qry_encoder.encoder.embeddings.position_ids error (#647)
• fix asr paraformer finetune bug
• fix uie trainer: eval failed (#617)
• Fix vllm: change if condition (#607)
• fix shop_segmentation to use old timm lib and bump version to 1.9.4rc2
• fix the numpy bug for card detection correction
• fix issues for 3dhuman models
• fix logger: remove file handler for original user logging (#645)

中文版本

Feature

• 新增句子向量模型，支持gte, bloom
• stable diffusion新增freeU方法
• LLMPipeline 支持Swift adapter模型推理
• 镜像制作时自动升级funasr transformer最新版本
• venv强制依赖移除，以便更好地支持windows系统 #575

bugfix

• 修复 shop_segmentation pipeline兼容timm 0.5.2
• 修复huggingface position_ids兼容性问题
• 修复chatglm sp_tokenizer属性确实问题
• 修复ofa模型transformers新版兼容性问题
• 修复trainer中work_dir设置不生效问题 #573
• 修复hf相关的bug #569 #567

新增模型推荐

序号	模型名称&快捷链接
1	GTE文本向量-中文-通用领域-large
2	GTE文本向量-英文-通用领域-large
3	GTE文本向量-英文-通用领域-small
4	GTE文本向量-英文-通用领域-base
5	GTE文本向量-中文-通用领域-small
6	X-vector说话人转换点定位-两人-中文
7	Udever 多语言通用文本表示模型 3b
8	Udever 多语言通用文本表示模型 1b1
9	GTE文本向量-中文-通用领域-base
10	基于扩散模型的人物多视图生成模型
11	Udever 多语言通用文本表示模型 560m
12	Udever 多语言通用文本表示模型 7b1
13	通义千问-14B-Chat-Int8
14	通义千问-7B-Chat-Int8
15	CT-Transformer标点-中英文-通用-large-onnx
16	CodeFuse-QWen-14B
17	ECAPA-TDNN说话人确认-中文-CNCeleb-16k
18	ECAPA-TDNN说话人确认-中文-3D-Speaker-16k
19	中文字体风格迁移模型
20	Whisper语音识别-英文-small
21	Whisper语音识别-多语言-large
22	中文字体生成基础模型
23	FreeU文本生成图像模型
24	Paraformer语音识别-英文-通用-16k-离线-长音频版
25	Paraformer分角色语音识别-中文-通用
26	Paraformer语音识别-英文-通用-16k-离线-large-onnx
27	PASDv2图像超分辨率
28	Transducer语音识别-英文-gigaspeech-16k-实时
29	PMR-base
30	PMR-large
31	EQA-PMR-large
32	CodeFuse-StarCoder-15B
33	基于NER微调的机器阅读理解模型
34	CodeFuse-CodeLlama-34B-4bits
35	零样本文本分类-SSTuning-base-多语
36	通义千问-14B-Chat-Int4
37	零样本文本分类-SSTuning-base-英语
38	人脸检测与五官定位
39	多语言Conformer Listener
40	SambertHifigan语音合成-多语言-多人预训练-16k
41	音频量化编码-freqcodec_magphase-英文-libritts-16k-gr8nq32ds320-pytorch
42	音频量化编码-freqcodec_magphase-英文-libritts-16k-gr1nq32ds320-pytorch
43	音频量化编码-Encodec-中英文-通用-16k-nq32ds640-pytorch
44	音频量化编码-Encodec-中英文-通用-16k-nq32ds320-pytorch
45	音频量化编码-Encodec-英文-libritts-16k-nq32ds320-pytorch
46	ERes2Net-Large说话人日志-对话场景角色区分-通用
47	3DHuman-Syn三维角色驱动
48	音频量化编码-Encodec-英文-libritts-16k-nq32ds640-pytorch
49	文本生成3D头部模型
50	文本引导模型纹理生成-三维视觉
51	3DHuman-Syn生成式3D人物模型库

English Version

Feature

• Added sentence vector model, supporting gte and bloom.
• Stable diffusion introduces a new freeU method.
• LLMPipeline now supports Swift adapter model inference.
• Automatically upgrade to the latest version of funasr transformer during image creation.
• Forced venv dependency removed to better support Windows system. #575

bugfix

• Fixed shop_segmentation pipeline compatibility with timm 0.5.2.
• Resolved compatibility issues with huggingface position_ids.
• Fixed the missing sp_tokenizer attribute in chatglm.
• Addressed compatibility issues of ofa model with newer transformers version.
• Resolved the issue where the work_dir setting in trainer was not taking effect. #573
• Fixed hf-related bugs. #569 #567.

New Models Recommended

No	Model Name & Link
1	nlp_gte_sentence-embedding_chinese-large
2	nlp_gte_sentence-embedding_english-large
3	nlp_gte_sentence-embedding_english-small
4	nlp_gte_sentence-embedding_english-base
5	nlp_gte_sentence-embedding_chinese-small
6	speech_xvector_transformer_scl_zh-cn_16k-common
7	udever-bloom-3b
8	udever-bloom-1b1
9	nlp_gte_sentence-embedding_chinese-base
10	multimodal_multiview_avatar_gen
11	udever-bloom-560m
12	udever-bloom-7b1
13	Qwen-14B-Chat-Int8
14	Qwen-7B-Chat-Int8
15	punc_ct-transformer_cn-en-common-vocab471067-large-onnx
16	CodeFuse-QWen-14B
17	speech_ecapa-tdnn_sv_zh-cn_cnceleb_16k
18	speech_ecapa-tdnn_sv_zh-cn_3dspeaker_16k
19	font_style_transfer_model
20	speech_whisper-small_asr_english
21	speech_whisper-large_asr_multilingual
22	font_generation_base_model
23	multi-modal_freeu_stable_diffusion
24	speech_paraformer-large-vad-punc_asr_nat-en-16k-common-vocab10020
25	speech_paraformer-large-vad-punc-spk_asr_nat-zh-cn
26	speech_paraformer-large_asr_nat-en-16k-common-vocab10020-onnx
27	PASD_v2_image_super_resolutions
28	speech_conformer_transducer_asr-en-16k-gigaspeech-vocab5001-pytorch-online
29	PMR-base
30	PMR-large
31	EQA-PMR-large
32	CodeFuse-StarCoder-15B
33	NER-PMR-Large
34	CodeFuse-CodeLlama-34B-4bits
35	zero-shot-classify-SSTuning-XLM-R
36	Qwen-14B-Chat-Int4
37	zero-shot-classify-SSTuning-base
38	cv_face_detection_landmark
39	speech_conformer_larger_asr_multi_language-16k-common-vocab30392-pytorch
40	speech_sambert-hifigan_tts_multilingual_multisp_pretrain_16k
41	audio_codec-freqcodec_magphase-en-libritts-16k-gr8nq32ds320-pytorch
42	audio_codec-freqcodec_magphase-en-libritts-16k-gr1nq32ds320-pytorch
43	audio_codec-encodec-zh_en-general-16k-nq32ds640-pytorch
44	audio_codec-encodec-zh_en-general-16k-nq32ds320-pytorch
45	audio_codec-encodec-en-libritts-16k-nq32ds320-pytorch
46	speech_eres2net-large_speaker-diarization_common
47	cv_3d-human-animation
48	audio_codec-encodec-en-libritts-16k-nq32ds640-pytorch
49	cv_HRN_text-to-head
50	cv_diffuser_text-texture-generation
51	cv_3d-human-synthesis-library

中文版本

高亮功能

• 优化ci
• 兼容transformers 4.34.0
• Support int4 model for llm_pipeline

BugFix

• fix merge error (#582)
• move venv import from file level to class level to avoid import error… (#575)

English Version

Highlight

• optimize ci
• compatible with transformers 4.34.0
• Support int4 model for llm_pipeline

BugFix

• fix merge error (#582)
• move venv import from file level to class level to avoid import error… (#575)

中文版本

新模型推荐

高亮功能

• 增加image_control_3d_portrait模型
• 增加3dhuman render and animation 模型
• 增加LLMPipeline支持大模型推理

功能列表

• 支持swift trainer和pipeline
• 增加image_control_3d_portrait模型
• 增加3dhuman render and animation 模型
• 新增 model for card correction
• 新增 head_reconstruction and text_to_head model
• 增加LLMpipeline支持大模型推理
• 增加 onnx exporter for ocr recognition model
• 增加 onnx exporter for ocr_detection db model

功能提升

BugFix

• 修复onnxruntime 新版本兼容性问题
• 修复huggingface兼容性问题
• asr支持本地模型
• 修复ci问题

English Version

New Model List and Quick Access

Highlight

• Add 3dhuman render and animation models
• Add image_control_3d_portrait
• Add LLMpipeline support LLM inference

Breaking changes

Feature

• support swift trainer and pipeline (#547)
• add image_control_3d_portrait
• add 3dhuman render and animation models
• add model for card correction
• add head_reconstruction and text_to_head model
• add LLMpipeline support LLM inference
• add onnx exporter for ocr recognition model
• add onnx exporter for ocr_detection db model

Improvements

BugFix

• Fix onnxruntime providers parameter compatible issue
• Fix hf bug (#569)
• Fix support local asr models (#556)
• Fix fix ci issue

中文版本

新模型推荐

高亮功能

• 模型下载增加失败重试功能

功能列表

功能提升

• 支持模型下载失败重试

BugFix

• 解决新版本transformers position_ids兼容性问题

English Version

New Model List and Quick Access

Highlight

• Retry download model when failed.

Breaking changes

Feature

• Retry download model when failed.

Improvements

BugFix

• Fix latest transformers position_ids compatible issue.

中文版本

新模型推荐

序号	模型名称&快捷链接
1	通义千问-VL-Chat-Int4
2	t5-base
3	WizardMath-7B-V1.0
4	WizardCoder-3B-V1.0
5	WizardCoder-Python-13B-V1.0
6	WizardCoder-Python-34B-V1.0
7	WizardMath-13B-V1.0
8	WizardLM-30B-V1.0
9	WizardLM-7B-V1.0
10	CodeLlama-34b-Instruct-hf
11	CodeLlama-34b-Python-hf
12	CodeLlama-13b-Instruct-hf
13	CodeLlama-7b-hf
14	CodeLlama-7b-Python-hf
15	CodeLlama-34b-hf
16	CodeLlama-13b-hf
17	CodeLlama-13b-Python-hf
18	CodeLlama-7b-Instruct-hf
19	WizardCoder-15B-V1.0
20	WizardCoder-1B-V1.0
21	WizardMath-7B-V1.0
22	WizardLM-13B-V1.2
23	ERes2Net-Base语种识别-中英粤日韩识别-8k
24	ERes2Net-large语种识别-中英粤日韩识别-8k
25	Paraformer语音识别-英文-通用-16k-离线-1B-pytorch
26	Regularized DINO说话人确认-中文-CNCeleb-16k

高亮功能

• video to video model support a10, v100
• funasr支持mossformer模型
• SDXL支持lora微调
• stable diffusion支持fp16训练和推理

不兼容更新

• 最低支持python3.8
• 镜像tensorflow 升级2.13.0
• numpy，pandas版本升级

功能列表

功能提升

• 更新qwen QA 示例
• 添加qwen QA langchain示例

BugFix

• 修复stable diffusion fp16 bug
• 修复图像上色模型加载问题
• 修复 chatglm2b rope_ratio config 缺失问题

English Version

New Model List and Quick Access

No	Model Name & Link
1	qwen-VL-Chat-Int4
2	t5-base
3	WizardMath-7B-V1.0
4	WizardCoder-3B-V1.0
5	WizardCoder-Python-13B-V1.0
6	WizardCoder-Python-34B-V1.0
7	WizardMath-13B-V1.0
8	WizardLM-30B-V1.0
9	WizardLM-7B-V1.0
10	CodeLlama-34b-Instruct-hf
11	CodeLlama-34b-Python-hf
12	CodeLlama-13b-Instruct-hf
13	CodeLlama-7b-hf
14	CodeLlama-7b-Python-hf
15	CodeLlama-34b-hf
16	CodeLlama-13b-hf
17	CodeLlama-13b-Python-hf
18	CodeLlama-7b-Instruct-hf
19	WizardCoder-15B-V1.0
20	WizardCoder-1B-V1.0
21	WizardMath-7B-V1.0
22	WizardLM-13B-V1.2
23	ERes2Net-Base
24	ERes2Net-large
25	Paraformer
26	Regularized

Highlight

• video to video model support a10, v100
• Add funasr support mossformer model
• Support sdxl finetune by lora method
• support float16 training and pipeline for stable diffusion

Breaking changes

• Deprecated python3.7 support
• tensorflow upgrade to 2.13.0 in image
• numpy, pandas version upgrade

Feature

Improvements

• upgrade qwen QA sample

BugFix

• Fix bugs of stable diffusion fp16
• fix image colorization model load issue
• fix chatglm2b rope_ratio config is missing

bugfix for qwen

• streamer pass error
• check flash attention installation even if use_fast_att is set True
• fix quantization model run failed

中文版本

新模型推荐

序号	模型名称&快捷链接
1	千问-7B  千问-7B-chat
2	chatglm2-6b-32k
3	MDQE视频实例分割
4	语音合成-越南语-通用领域-24k-发音人tien
5	语音合成-马来语-通用领域-24k-发音人farah
6	stable-diffusion-xl-refiner-1.0
7	stable-diffusion-xl-base-1.0
8	PolyLM-智能服务-文本生成模型-多语言-13B
9	鹏城·盘古增强版-2.6B-CPU
10	ERes2Net-Base语种识别-中英文识别-16k
11	ERes2Net-Large语种识别-中英文识别-16k
12	codegeex2-6b
13	openbuddy-llama2-13b-v8.1-fp16
14	CT-Transformer标点-中英文-通用-large
15	CAM++语种识别-中英文识别-16k
16	zeroscope_v2_xl高清文生视频
17	FreeWilly2
18	Beautiful-Realistic-Asians-v5
19	ProST: 视频文本通用检索模型
10	Realistic_Vision_V4.0
21	CAM++说话人确认-中文-3DSpeaker-16k
22	Llama-2-70b-ms
23	Llama-2-13b-chat-ms
24	Llama-2-7b-ms
25	Llama-2-7b-chat-ms
26	Llama-2-13b-ms
27	PolyLM-指令精调-文本生成模型-多语言-13B
28	LLaVA视觉问答模型
29	Paraformer语音识别-英文-通用-16k-离线-large-pytorch
30	speech_bert_semantic-spk-turn-detection-punc_speaker-diarization_chinese
31	生成扩散模型高效调优-Swift-LoRA
32	生成扩散模型高效调优-Swift-Adapter
33	生成扩散模型高效调优-Swift-Prompt
34	MindChat-7B
35	MindChat-6B
36	MindChat-Baichuan-13B
37	rwkv-4-music
38	RWKV-4-Raven-7B
39	rwkv-4-world
40	球面上的全景图单目深度估计
41	AquilaChat-7B
42	Sunsimiao-6B-05M
43	Sunsimiao-InternLM-01M
44	4K 超高清 NeRF 重建算法
45	基于扩散模型的文生图-360全景图生成模型
46	speech_bert_dialogue-detetction_speaker-diarization_chinese
47	PolyLM-文本生成模型-多语言-13B
48	stable-diffusion-xl-base-0.9
49	百川13B对话模型
50	百川13B模型
51	书生·浦语大模型
52	internlm-chat-7b
53	StableSR图像超分辨率
54	ERes2Net-Large说话人确认-中文-3D-Speaker-16k
55	ERes2Net-Base说话人确认-中文-3D-Speaker-16k
56	基于向量量化的神经辐射场压缩
57	baichuan_agent
58	Regularized DINO说话人确认-中文-3D-Speaker-16k
59	CAAI-Hackathon
60	BAT语音识别-中文-aishell1-16k-离线

功能列表

• 训练时使用AutoModel情况下增加模型版本检查
• 支持safe tensors weight pipeline。
• 支持transformers类模型的流式输出。
• 优化类包装器。
• 支持huggingface transformers的AutoModel, AutoConfig and AutoTokenizer
• 在llm中添加了完整参数的sft。
• 添加了日语README。
• 在params和load函数中添加了download_mode参数。
• 使用Xformers 提高了attention计算时的内存/显存性能。

功能提升

• 将 stable diffusion 版本升级到更强大的版本2.1。
• 增加custom stable diffusion 微调。
• 添加了 stable diffusion swift tuner。
• 在不使用deepspeed的情况下支持llama和lora微调。
• 为lora  stable diffusion 添加了lora_rank参数。
• 优化了torch1.11和torch2.0.1镜像构建脚本。
• 在sbert文本分类中支持从数据集获取标签，并在chatglm-6b中构建文件数据集。
• 更新了speaker_verification_pipeline.py。
• 更新了默认测试级别。
• 更新了aliyuncs pip的默认值。
• 更新了语言识别任务名称。
• 为text-to-video添加了height和width参数。
• 自定义 diffusion pipeline。
• 添加了支持ASRDataset的download_mode参数。
• 更新了chatglm6b v2的新版本。
• 改进了加载meta-csv缓存路径的方式。
• 添加了一个example/llm模块。
• 优化了注释和格式。
• 设置download_mode的默认值。
• 使用num_inference_steps和guidance_scale参数更新pipeline。
• 在finetune_speech_recognition.py中添加了download_mode的支持，使用params.download_mode。
• 在finetune_speech_recognition.py中使用ASRDataset替换MsDataset。
• 更新了ASRDataset，为重新下载数据集（如数据集损坏或损坏）添加了download_mode。
• 更新了asr_dataset.py，支持使用download_mode重新下载数据。
• 将text_in设置为必需参数。
• 修改了text_generation_pipeline类的参数传递。
• 添加了baichuan/chatglm2+lora+agent示例。
• 添加了 stable diffusion 教程ipynb。

BugFix

• 忽略http错误，以防止模型检查时混淆。
• 解决了加载 checkpoint时出现的不同设备问题。
• 修复了缺少plugin模块文件的问题。
• 解决了镜像标签无cuda的问题。
• 解决了easycv CPU扩展构建问题。
• 解决了设备错误问题。
• 解决了amp和device_map问题。
• 通过配置pysptk >= 0.1.19解决了pip安装错误。
• 解决了ckpt输出目录忽略*.safetensors的问题。
• 修复了对baichuan的eval和sequence_length支持的问题。
• 解决了在文档分割pipeline中使用cuda设备的问题。
• 删除了清华相关的硬编码。
• 解决了与stable diffusion pipeline相关的错误，该pipeline无法识别“lora_scale”参数。
• 解决了与chatglm2模块相关的错误。
• 修复了mPLUG-Owl生成长度错误。
• 更正了speaker模型的详细信息。
• 修复了在加载本地stable diffusion数据集时出现的错误。
• 修复了chatglm管道中的错误。
• 修复了chatglm6b 2的错误。
• 修复了与empty hypothesis相关的chatglm2评估错误。

English Version

New Model List and Quick Access

No	Model Name & Link
1	Qwen-7B  Qwen-7B-chat
2	chatglm2-6b-32k
3	MDQE video-instance-segmentation
4	speech_sambert-hifigan_nsf_tts_tien_Vietnamese_24k
5	speech_sambert-hifigan_nsf_tts_farah_Malay_24k
6	stable-diffusion-xl-refiner-1.0
7	stable-diffusion-xl-base-1.0
8	PolyLM-assistant_13b_text_generation
9	pangu-plus-2.6B-CPU
10	ERes2Net-Base-language identification-en-cn-16k
11	ERes2Net-Large-language identification-en-cn-16k
12	codegeex2-6b
13	openbuddy-llama2-13b-v8.1-fp16
14	CT-Transformer-punc-cn-en-common-large
15	CAM++-language identification-en-cn-16k
16	zeroscope_v2_xl high-definition text-to-video generation
17	FreeWilly2
18	Beautiful-Realistic-Asians-v5
19	ProST: retrieval model for video-text
20	Realistic_Vision_V4.0
21	CAM++-zh-cn-3DSpeaker-16k
22	Llama-2-70b-ms
23	Llama-2-13b-chat-ms
24	Llama-2-7b-ms
25	Llama-2-7b-chat-ms
26	Llama-2-13b-ms
27	PolyLM-multialpaca-text_generation-13B
28	LLaVA visual-question-answering
29	Paraformer-asr_nat-zh-cn-16k-large-pytorch
30	speech_bert_semantic-spk-turn-detection-punc_speaker-diarization_chinese
31	multi-modal_efficient-diffusion-tuning-Swift-LoRA
32	multi-modal_efficient-diffusion-tuning-Swift-Adapter
33	multi-modal_efficient-diffusion-tuning-Swift-Prompt
34	MindChat-7B
35	MindChat-6B
36	MindChat-Baichuan-13B
37	rwkv-4-music
38	RWKV-4-Raven-7B
39	rwkv-4-world
40	monocular depth estimation for panoramic images on a sphere
41	AquilaChat-7B
42	Sunsimiao-6B-05M
43	Sunsimiao-InternLM-01M
44	4K Ultra high definition NeRF 3d-reconstruction
45	diffusion_text-to-360panorama-image_generation
46	speech_bert_dialogue-detetction_speaker-diarization_chinese
47	PolyLM-text_generation-13B
48	stable-diffusion-xl-base-0.9
49	Baichuan-13B-Chat
50	Baichuan-13B-Base
51	internlm-chat-7b-8k
52	internlm-chat-7b
53	StableSR image-super-resolution
54	ERes2Net-Large large_sv_zh-cn_3dspeaker_16k
55	ERes2Net-Base base_sv_zh-cn_3dspeaker_16k
56	3d-reconstruction_vector-quantize-compression
57	baichuan_agent
58	Regularized DINO ecapa_tdnn -3D-Speaker-16k
59	CAAI-Hackathon
60	BAT-asr-zh-cn-aishell1-16k

Feature

• Added check model for training and model_dir for automodel.
• Added support for safetensors weight pipeline.
• Support for stream output on transformers model
• Refined class wrapper
• Added Support for AutoModel, AutoConfig and AutoTokenizer
• Added full parameter sft to llm
• Added Japanese README
• Added download_mode param to params and load function.
• Accelerated memory efficient attention with Xformers.

Improvements

• Upgraded stable diffusion version to more powerful version 2.1.
• Custom method for finetuning stable diffusion
• Added stable diffusion swift tuner
• Support for llama & lora finetune without deepspeed
• Added lora_rank parameter for lora stable diffusion
• Refactored torch1.11 and torch2.0.1 build script
• Support getting labels from dataset in sbert text classification and building dataset from file in chatglm-6b
• Updated speaker_verification_pipeline.py
• Changed tests level
• Set default aliyuncs pip
• Added height and width parameters for text-to-video.
• Customized the diffusion pipeline.
• Added support for ASRDataset for download_mode parameters.
• Updated the Fea/chatglm6b v2 new version.
• Improved the load meta-csv cathe paths.
• Added an example/llm module.
• Optimized comments and formatting.
• Set the download_mode default value.
• Updated the pipeline with num_inference_steps and guidance_scale parameters.
• Added support for download_mode in finetune_speech_recognition.py with params.download_mode.
• Replaced MsDataset with ASRDataset in finetune_speech_recognition.py.
• Updated ASRDataset with download_mode for re-downloading the dataset if it is broken or corrupted.
• Updated asr_dataset.py to support download_mode for re-downloading data.
• Made text_in a required parameter.
• Modified the parameter passing of the text_generation_pipeline class.
• Added baichuan/chatglm2 +lora+agent examples.
• Added stable diffusion tutorial ipynb.

BugFix

• Ignored http error to prevent confusion during model check.
• Fixed checkpoint issue related to same device.
• Fixed missing plugin python module files.
• Fixed build tag no cuda issue.
• Fixed easycv CPU extension build issue.
• Fixed device error.
• Fixed issue with amp and device_map
• Fixed pip install error with pysptk>=0.1.19
• Fixed issue with ckpt output directory ignoring *.safetensors
• Fixed eval and sequence_length support for baichuan
• Fixed issue with using cuda device in document segmentation pipeline inference
• Removed hard code tsinghua
• Fixed amp + device_map (#386)
• Fixed bugs related to the Chinese stable diffusion pipeline not recognizing the 'lora_scale' argument.
• Fixed bugs related to the chatglm2 module.
• Fixed mPLUG-Owl generating length bug.
• Fixed details of speaker models.
• Fixed bugs related to loading local sd dataset.
• Fixed bugs in the chatglm pipeline.
• Fixed chatglm6b 2.
• Fixed chatglm2 evaluation error related to empty hypothesis.

中文版本

新特性

• 增加baichuan模型lora inference
• 增加baichuan和chatglm2 lora agent示例

BugFix

• 修复历史问题
• 修复sd加载本地数据集的问题
• 修复chatglm推理问题
• 修改text_generation_pipeline传参
• 修复chatglm6b和chatglm6b 2
• 修复评估报错：hypothesis emtpy

English Version

Features

• Add lora_inference for baichuan
• Add baichuan/chatglm2+lora+agent examples

BugFix

• fix history problem
• Fix a bug of loading local stable diffusion dataset
• Fix/chatglm pipeline
• Modify the parameter passing of the text_generation_pipeline class
• Fix/chatglm6b 2
• Fix/chatglm6b
• fix evaluation error: hypothesis emtpy

中文版本

新模型推荐

序号	模型名称&快捷链接
1	读光-文字识别-轻量化端侧识别模型-中英-通用领域
2	读光-文字检测-轻量化端侧DBNet行检测模型-中英-通用领域
3	CAM++说话人转换点定位-两人-中文

高亮功能

• 新增轻量化端侧识别模型LightweightEdge
• 新增轻量化端侧DBNet行检测模型
• 新增CAM++说话人转换点定位
• llama模型支持finetune和deepseed
• llama模型支持lora
• 对于transformers模型支持device_map
• 数据集支持jsonl格式
• 对于大型模型文件支持并行下载（dsw或eais环境）
• 提升youku超大型数据集下载体验

功能列表

• 新增轻量化端侧识别模型LightweightEdge
• 新增轻量化端侧DBNet行检测模型
• 新增flextrain ner样例
• 新增在文本分类finetune的training args中增加模型版本
• 新增StreamingMixin
• 支持torch extension
• 支持llama模型微调和deepspeed
• pipeline中支持第三方的key
• 支持说话人分离pipeline
• 新增eres2net_aug v2模型
• 支持transformers model device_map
• 支持模型权重diff
• 数据集支持jsonl格式
• 新增Lora/Adapter/Prompt/Chatglm6b
• 部分tests增加teardown
• 解除datasets包版本限制
• 支持从model id加载
• llama模型支持lora
• 对于大型模型文件支持并行下载

功能提升

• 提升mPLUG-youku大型数据集的下载体验

BugFix

• 修复DeepspeedHook.register_processor
• dockerfile的兼容性修改（py37和py38）
• 修复 extra_args
• 修复ngpu bug和移除easyasr
• 修复mplug-youku超大数据集下载相关问题
• 修复gpt3 finetune nan的问题
• 修复torch extension ci hang住的问题
• 修复easycv lr hook 错误
• 修复torch2.x 兼容性问题
• 修复diffusers版本冲突问题
• 修复eval RecursionError
• 对于DiffusionForTextToImageSynthesis修复device_map问题
• 修复stable diffusion pipeline cpu推理问题
• 修复llama lora问题

English Version

New Model List and Quick Access

No	Model Name & Link
1	cv_LightweightEdge_ocr-recognitoin-general_damo
2	cv_proxylessnas_ocr-detection-db-line-level_damo
3	speech_campplus-transformer_scl_zh-cn_16k-common

Highlight

• Add new OCR recognition model (LightweightEdge) and some functions
• Add ocr detection new model db-nas
• Add CAM++ model
• Support llama model finetune and deepspeed
• Support lora for llama model
• Support device_map for transformers
• Support jsonl format in datasets
• Support parallel download large model file
• Improve mPLUG-YOUKU dataset downloading experience

Breaking changes

Feature

• Add new OCR recognition model (LightweightEdge) and some functions
• Add ocr detection new model db-nas
• Add ner example for flextrain
• Add model revision in training_args and modify dataset loading in finetune text classification
• Add StreamingMixin
• Support pre build torch extension build image, first extension megatron_util
• Add llama finetune + deepspeed
• Support third_party key in pipeline
• Add speaker diarization pipeline and improve some speaker pipelines
• Add eres2net_aug v2
• Support device_map for transformers model
• Add make diff & recover for model weights
• Support jsonl format in meta data
• Add Lora/Adapter/Prompt/Chatglm6b
• Add teardown for tests
• Unfreeze datasets version setting
• Support load from model id
• Support lora for llama
• Support parallel download large model file

Improvements

• llama tuned model -> pipeline
• improve youku dataset downloading experience

BugFix

• Fix bug for DeepspeedHook.register_processor
• Docker file py38 and py37 compatible merge
• Fix extra_args
• ngpu bug and rm easyasr
• Fix issues for downloading mplug-youku dataset
• Fix gpt3 finetune nan
• Fix ci hang when build torch extension
• Fix easycv lr hook error
• Fix torch 2.x compatible issue
• Fix diffuser version conflict cv and multi-modal
• Fix eval RecursionError
• Fix device_map for DiffusionForTextToImageSynthesis
• Fix cpu inference for stable diffusion pipeline
• Fix llama lora bug

中文版本

功能列表

• 支持跳过easycv三方依赖引入
• 支持Flextrain training args和push_to_hub
• 支持domain_specific_object_detection 的onnx格式导出

BugFix

• 修复test_cli CI报错
• 修复merge hook
• 修复NER tokenizer不能接收kwargs的问题
• 修复lineless_table_recognition功能遇到空白图片崩溃的bug
• 修复某些情况下private数据集鉴权失败的问题

English Version

Feature

• Add pattern to skip easycv.thirdparty
• Support flex train feature (training args and push_to_hub adaptions)
• Support onnx export for domain_specific_object_detection

BugFix

• Fix CI: test merge dataset failed
• Fix merge_hook
• Fix NER tokenizer which won't accept kwargs
• Fix lineless_table_recognition crashed when input blank images
• fix private dataset auth issue

中文版本

该版本共新增上架5个模型。

新模型列表及快捷访问

贡献组织	模型名称	是否支持Finetune
达摩院	ERes2Net说话人确认-英文-VoxCeleb-16k-离线-pytorch	否
达摩院	mPLUG-Owl-多模态对话-英文-7B	否
达摩院	FastInst快速实例分割	否
达摩院	TransFace人脸识别模型	否
达摩院	Regularized DINO说话人确认-英文-VoxCeleb-16k-离线-pytorch	否

非兼容性修改

• 支持Python3.8版本
• 移除demo check

English Version

Highlight

• Support Python3.8
• Add mPLUG-Owl model
• Add cvpr23 Fastinst model

Breaking changes

• Support Python3.8
• Remove demo check

Feature

• Add ERes2Net for speaker verification
• Add mPLUG-Owl model
• Support FlexTrain and update the structure of trainer
• Add cvpr23 fastinst model
• Support Virgo MaxCompute datasource for Ali-cloud inner applications
• Add clip_interrogator
• Add gpt3 example
• Add convert megatron ckpt script
• Add trainer for UniTE
• Add transface model
• Add verified if whl installed
• Support python3.8
• Add ONNX exporter for ans dfsmn
• Add rdino model

Improvements

• Update multi_modal_embedding example
• Refine easrasr
• Pipeline input, output and parameter normalization.
• Display hub error message
• Remove easycv codes, plugin access

BugFix

• Fix bug in **kwargs duplicated for audio module
• Fix distributed hook to lazyimport and an import bug
• Fix transformer examples
• Add pop for base class parameters
• Fix func update_local_model; change funasr version
• Remove pai-easycv requirement
• Fix hypotheses did't init in cpu device, make fid_dialogue_test available

中文版本

新模型推荐

序号	模型名称&快捷链接
1	ResNet50行人结构化属性识别模型
2	DamoFD人脸检测关键点模型-0.5G
3	CAM++说话人确认-英文-VoxCeleb-16k
4	一种具有自我评估能力的机器翻译-中英-通用领域-large

高亮功能

• 支持 lora 生成扩散模型高效调优
• 增加 llama 模型
• 支持推送到 hub 的能力
• 为 chatglm-6B 类模型支持 chat 任务
• 增加常用模型和任务的 cli 调用 example

功能列表

• 支持了对使用 megatron tensor 并行模型保存的 checkpoint 拆分合并
• 支持 lora 生成扩散模型高效调优
• 增加 pedestrian attribute recognition 模型
• 增加 damofd 系列模型
• 增加 llama 模型
• 支持推送到 hub 的能力
• 增加 speaker cam++ 模型
• 增加 head 支持 XlmRoberta 模型
• 增加 canmt translation 模型
• 为 chatglm-6B 类模型支持 chat 任务

功能提升

• funasr 更新到 0.4.0 版本，支持 mac 运行
• plugin 支持 trainer
• fid_dialouge_pipeline 新增 3.7B 模型
• 增加 Mgeo 模型 token classification 任务的训练示例
• 增加 PALM 模型 text generation 任务的训练示例
• 增加 CLIP 模型 multi-modal embedding 任务的训练示例
• speech kws nearfield 训练增加梯度累积配置
• 重构优化人脸重建模型相关代码
• 更新图像着色指标
• 更新 github issue 模版

BugFix

• 修复文本生成任务模型 generate 报错
• 修复人脸重建模型 pipeline 报错
• 修复 pipeline 重复输出 warning 的问题
• 修复 plugin import 包失败时报错
• 修复 speech kws nearfield 多卡训练报错
• 修复生成模型输出英文结果缺少空格的问题
• 修复 jsonplus 不支持 ndarray 的问题

English Version

New Model List and Quick Access

No	Model Name & Link
1	ResNet50 pedestrian-attribute-recognition image
2	DamoFD face-detection 0.5G
3	Speech cam++ English-VoxCeleb-16k
4	Canmt translation with self evaluation zh2en-large

Highlight

• Add efficient tunner modules
• Add llama to mslib from hf
• Support the ability to push to hub
• Add task chat for all chat models, like chatglm-6B
• Add common models and tasks cli call example

Breaking changes

Feature

• Support split and merge for megatron_base model
• Add efficient tunner modules
• Add pedestrian attribute recognition model
• Add damofd model
• Add llama to mslib from hf
• Support the ability to push to hub
• Add speaker model cam++ for speaker verification task
• New head support for XlmRoberta model
• Add canmt translation model
• Add task chat for all chat models, like chatglm-6B

Improvements

• support funasr for mac
• Plugin support trainer
• Add 3.7B size model for fid_dialouge_pipeline
• Add token classification example for MGeo
• Add PALM finetune example
• Add multi-modal embedding example for CLIP
• Speech kws nearfield training add gradient accumulation config
• Update face reconstruction to HRN(CVPR2023)
• Update image colorization metric
• Update issue templates

BugFix

• Fix generate for ModelForTextGeneration
• Fix issues for face pipeline
• Fix keep printing warnings in pipeline
• Bug fixed in plugin
• Fix speech kws nearfield training with multi-gpu
• Fix english words without space
• Fix jsonplus, support ndarray

中文版本

新模型推荐

序号	模型名称&快捷链接	贡献组织	是否支持finetune
1	ChatGLM-中英对话大模型-6B	智谱.AI
2	GLM130B-中英大模型	智谱.AI
3	unidiffuser-v1	清华TSAIL
4	元语功能型对话大模型v2	元语智能
5	盘古α 2.6B	鹏城实验室
6	openjourney	个人开发者-dienstag
7	Rwkv-4-pile-14b	个人开发者-Blink_DL
8	SiameseUIE通用信息抽取-中文-base
9	SiameseUniNLU零样本通用自然语言理解-中文-base · 模型库 (modelscope.cn)

高亮功能

• 外部repo可以以插件形式和modelscope库协同工作
• 增加SCRFD模型的onnx导出
• 增加damoyolo模型的onnx导出
• 支持序列标注模型的onnx/torchscript导出
• 支持cartoon模型的pb文件导出
• 重构taskdataset模块，用户现在可定制自己的数据集逻辑了
• 增加text-generation任务的examples，同样适用于GPT3
• Siamese uie模型支持finetune

功能列表

• 推理和训练中支持torch2.0 compile，注意因为测试尚不充分因此有些模型可能遇到错误
• Add adadet库的trainer支持
• ddcolor image colorization支持训练
• 增加video_instance_segmentation推理能力
• 增加CLI工具的插件能力
• 增加human reconstruction任务
• 增加vidt模型
• 增加speech_timestamp任务
• 增加disco guided diffusion模型
• ocr_reco_crnn支持训练
• 增加action detection的训练
• 增加ocr_detection_db的训练
• 增加 lore lineness table recognition任务
• 增加PEER模型
• 增加damoyolo的烟雾探测模型
• 增加RLEG模型
• 增加用于视频实例追踪的ProContEXT模型
• 增加视频感知模型longshortnet
• 增加dingding去噪模型
• 支持vision efficient tuning
• 支持text-to-video-synthesis任务

功能提升

• text-generation推理支持args输入
• 在video temporal grounding中支持soonet
• trainer支持DDPHook
• Kws支持继续训练能力
• 支持GPU上的正确DDIM采样能力
• 增加更多的CLI工具
• 修改语音推理的输入输出
• 优化kws的配置
• ImagePaintbyexamplePipeline支持demoservice
• 支持easycv trainer的load_from

BugFix

• 修复安装detecron2的报错
• 修复generate_scp_from_url方法的报错
• 修复speaker_verification_pipeline和speaker_diarization_pipeline
• 修复data releate case失败的问题
• 修复ast扫描失败的问题
• 修复Word alignment预处理器的bug

English Version

New Model List and Quick Access

No	Model Name & Link	Org	Finetune supported
1	ChatGLM-English&Chinese-6B	ZhiPu.AI
2	GLM130B-LLM English&Chinese	ZhiPu.AI
3	unidiffuser-v1	TsingHua TSAIL
4	ChatYuan-large-v2	YuanYu
5	OpenICommunity/pangu_2_6B	PengCheng Lab
6	openjourney	personal-dienstag
7	Rwkv-4-pile-14b	personal-Blink_DL
8	SiameseUIE information extraction-Chinese-base
9	SiameseUniNLU zero-shot NLU Chinese base model

Highlight

• Support repos work with modelscope library via plugin
• Support onnx export for SCRFD model
• Add onnx exporter for damoyolo
• Add onnx/torchscript exporter for token classification models
• Add frozen graph def exporter for cartoon model
• Refactor taskdataset module, user now can write datasets with custom logics
• Add example for text-generation finetuning, also available for GPT3
• Siamese uie finetune support

Breaking changes

Feature

• Support torch2.0 compile in inference and training, this feature is not stable on all models
• Add ADADET && thirdparty arg for damoyolo trainer
• Add finetune for ddcolor image colorization
• Add video_instance_segmentation pipeline
• Add plugin with cli tool
• Add human reconstruction task
• Add vidt model
• Add task: speech_timestamp
• Add disco guided diffusion
• Add training support for ocr_reco_crnn
• Add action detection finetune
• Add ocr_detection_db training module
• Add lore lineness table recognition
• Add PEER model
• Add smoke and fire detection model using damoyolo
• Add generative multimodal embedding model RLEG
• Add vop_se for text video retrival
• Add ProContEXT model for video single object tracking
• Add video streaming perception models longshortnet
• Add dingding denoise model
• Support vision efficient tuning finetune
• Add text-to-video-synthesis
• Add MAN for image-quality-assessment

Improvements

-Support run text generation pipeline with args

• Add soonet for video temporal grounding
• Trainer support parallel_groups setting and DDP hook
• Kws support continue training from a checkpoint
• Correct DDIM sampling on GPU
• Add more cli tools
• Modify audio input types && punc postprocess
• Optimize kws pipeline and training conf
• Support ImagePaintbyexamplePipeline demo service
• Support load_from for easycv trainer

BugFix

• Fix bug for install detecron2
• Fix bug for modify function generate_scp_from_url
• Fix bug for speaker_verification_pipeline and speaker_diarization_pipeline: re-write the default config with configure.json
• Fix bug for data releate case failed bug
• Fix bug for ast scan funcitondef
• Word alignment preprocessor fix

中文版本

新模型列表及快捷访问

该小版本共新增上架6个模型，其中新增2个模型支持finetune能力。

序号	模型名称&链接	支持finetune
1	ControlNet可控图像生成
2	兰丁宫颈细胞AI辅助诊断模型
3	读光-文字检测-DB行检测模型-中英-通用领域
4	SOND说话人日志-中文-alimeeting-16k-离线-pytorch
5	NeRF快速三维重建模型	√
6	DCT-Net人像卡通化	√

Feature

• GPT3 Finetune功能完善，支持DDP+tensor parallel， finetune流程串接推理流程优化
• checkpoint保存逻辑优化，确保周期性保存和最优保存的文件可以直接用于推理
• Hooks方案重构，解耦各个功能hook，支持hooks间交互
• 支持ImagePaintbyExamplePipeline demo service
• 支持多种音频类型
• 支持Petr3D CPU推理支持兼容新版mmcv
• deberta v2 预处理器更新
• 支持NLP下游任务模型初始化仅加载backbone预训练权重
• 更新librosa.resample()参数支持最新版本
• 添加下游工具箱调用埋点统计功能

不兼容行问题

• checkpoint保存分拆了模型参数和训练状态参数，老版本的模型参数需要转换后加载

问题修复：

• 修复asr vad/lm/punc输入处理
• 修复gpt moe finetune checkpoint path error
• 修复args lm_train_conf is invalid
• 修复删除已有文件ci测试报错
• 修复OCR识别bug
• 移除preprocessing stage中图像分辨率的限制
• 修复输出wav文件是32-bit float而不是预期的16-bit int
• 设置num_workers=0，以防止在demo-service中创建子进程

English Version

New Model List and Quick Access

This minor version adds a total of six new models, including two models with finetuning capability.

No.	Model Name & Link	Finetuning Supported
1	ControlNet Controllable Image Generation
2	Landing AI Cervical Cell AI-assisted Diagnosis Model
3	Reading Light - Text Detection - DB Row Detection Model - Chinese and English - General Domain
4	SOND Speaker Diary - Chinese - Alimeeting-16k - Offline - PyTorch
5	NeRF Fast 3D Reconstruction Model	√
6	DCT-Net Person Image Cartoonization	√

Features

• GPT-3 finetune has been improved to support DDP+tensor parallel
• Checkpoint saving logic has been optimized to ensure that files saved periodically and those saved as the best can be used directly by pipeline
• The Hooks scheme has been refactored to decouple various functional hooks and support interaction between hooks.
• Supports ImagePaintbyExamplePipeline demo service
• Supports multi-machine data and tensor parallel finetuning for cartoon task
• Supports various audio types
• Supports Petr3D CPU inference with compatibility for the latest version of mmcv
• Updates deberta v2 preprocessor
• Supports initialization of downstream NLP task models with only backbone pre-training weights loaded
• Updates librosa.resample() parameter support to the latest version
• Adds downstream toolbox call tracking function

Break changes

• Saving model parameters and training state seperately, so previous trained checkpoints should be converted before resume training

Bug Fixes:

• Fixes asr vad/lm/punc input processing
• Fixes gpt moe finetune checkpoint path error
• Fixes args lm_train_conf is invalid
• Fixes ci test errors when deleting existing files
• Fixes OCR recognition bugs
• Removes image resolution restrictions in preprocessing stage
• Fixes output wav file being 32-bit float instead of expected 16-bit int
• Sets num_workers=0 to prevent creating sub-processes in demo-service.

中文版本

该版本共新增上架51个模型，其中11个模型支持finetune能力。

模型功能特性说明

• 提供finetune的示例脚本，允许用户通过运行脚本命令行传参方式进行模型训练，详细可以参考github脚本

• NLP领域新增了backbone + head的开发支持，允许用户任意组合已有的backbone(Encoder) 和任务head，方便在特定任务上切换不同模型进行建模，详细参考文档

• 贡献者文档完善模型贡献部分，详细参考接入流程概览

• 数据集接口支持本地文件直接加载  MsDataset.load('/to/path/abc.csv')

• 模型导出支持nlp_structbert_zero-shot、 nlp_csanmt_translation系列模型

更多SDK功能和变更可查看：https://github.com/modelscope/modelscope/releases/tag/v1.3.0

新模型列表及快捷访问

序号	模型名称&链接	支持finetune
1	NAFNet图像去模糊	√
2	BEiTv2图像分类-通用-base	√
3	BEiTv2图像分类-通用-large	√
4	实时人头检测-通用	√
5	实时手机检测-通用	√
6	NAFNet图像去模糊压缩	√
7	DINO-高精度目标检测模型	√
8	StructBERT文本相似度-中文-电商-base	√
9	StructBERT事实准确性检测-中文-电商-base	√
10	StructBERT FAQ问答-中文-金融领域-base	√
11	StructBERT FAQ问答-中文-政务领域-base	√
12	IR人脸识别模型FRIR
13	口罩人脸识别模型FRFM-large
14	人脸质量模型FQA
15	静默人脸活体检测模型-炫彩
16	运动生成-人体运动-英文
17	M2FP单人人体解析
18	DeOldify视频上色
19	图像质量MOS评估
20	异常图像检测
21	YOLOPV2车辆检测车道线分割-自动驾驶领域
22	DCT-Net人像卡通化-扩散模型-插画
23	DCT-Net人像卡通化-扩散模型-漫画
24	卡通系列文生图模型
25	卡通系列文生图模型-漫画风
26	卡通系列文生图模型-水彩风
27	卡通系列文生图模型-剪贴画
28	卡通系列文生图模型-扁平风
29	轻量级SRResNet视频超分辨率
30	ECBSR端上图像超分模型
31	实时交通标识检测-自动驾驶领域
31	多尺度局部平面引导的单目深度估计
33	uhdm图像去摩尔纹
34	M2FP多人人体解析
35	VFI-RAFT视频插帧-应用型
36	StableDiffusionV2图像填充
37	MT5开放域多轮对话改写-中文-通用-base
38	基础视觉模型高效调优-adapter
39	基础视觉模型高效调优-prompt
40	基础视觉模型高效调优-prefix
41	基础视觉模型高效调优-lora
42	视频全景分割-VideoKNet-SwinB
43	人脸重建模型
44	DDPM-Seg基于扩散模型的语义分割
45	DeepLPF图像调色
46	视频去场纹
47	Adaptive-Interval-3DLUT图像调色
48	RealESRGAN图像去色带
49	图像画质损伤分析
50	基于视觉和语言的知识蒸馏的开放词汇目标检测
51	针对长尾/小目标问题的高性能通用目标检测

最佳实践教程

最后，我们还推出许多任务级别和模型级别的最佳实践教程文档，旨在帮助开发者更好地理解和应用模型。

• NLG大模型的使用介绍

• 序列标注任务最佳实践

• 文本生成任务最佳实践

• 文本分类任务最佳实践

• 图像分割任务最佳实践

欢迎关注我们的开源社区：https://github.com/modelscope/modelscope

English Version

Highlight

• Add vqa-degradation
• Add content check pipeline
• Add pipelines for en2zh-imt and zh2en-imt
• Add single and multiple human parsing models
• Add AdaInt model
• Add open vocabulary detection
• Support finetune for sentence-embedding
• Add bad image detection model and pipeline
• Support translation model exporting
• Add asr dataset for finetune
• Add ocr detection model and pipeline
• Add face quality assessment model
• Add video deinterlace model
• Add language model for audio task
• Add deeplpf for image color enhance and image debanding model
• Add ecbsr model for mobile image super-resolution
• Add msrresnetlite model for video super-resolution
• Support finetune and evaluation for image-fewshot-detection-defrcn
• Add yolopv2 model cv_yolopv2_image_driving_perception
• Add face liveness xc model
• Add paint-by-example model
• Add universal_matting pipeline
• Add multi-modal_gridvlp_classification_chinese-base-ecom-cate
• Add DINO detection with easycv
• Add speech speaker verification pipeline
• Add nerf-recon model
• Support finetune for real-time object detection with easycv
• Add single-camera depth estimation bts model
• Add MGIMN model
• Add fuse-in-decoder dialogue task
• Add vision_efficient_tuning models
• Add traffic-sign detection
• Add object_detection3d_depe model
• Add stable diffusion model for image inpainting
• Add head&phone detection models
• Add face_reconstruction model
• Add structured model probing pipeline for image classification
• Add video panorama segmentation with VideoKNet-SwinB
• Add image quality assessment mos(mean option score) model
• Add ddpm-segmentation pipeline
• Add plug mental model
• Add video-colorization pipeline
• Add image demoireing
• Add face recognition ir model
• Support batch inference for nlp_csanmt_translation_en2zh
• Add image_deblurring_dataset for REDS dataset
• Add new motion-generation model
• Add face recognition and face mask model

Breaking changes

• Adjust video_multi_target_tracking output
• Adjust video_human_matting output of video to support demo service

Feature

• Add default preprocessor for taskmodels
• Run ci cases base on code diff to reduct ci test time
• Support demo code to return path of result video for video human matting
• Add en2ru and ru2en pipeline ut v4
• Kws pipeline returns Chinese charactor by configuration
• Ast-scanning skip function level imports index
• Compatible with diffusers0.12.1
• Video depth estimation support cpu mode
• asr pipeline add output_dir parame
• Add RTS face recognition ood model
• Add image-defrcn-fewshot-detection

Improvements

• Remove requirements of mpi4py
• Remove pytorch-lightning version constrain
• Refine cv_image_defrcn trainer to avoid failed
• Support trainer prediction
• Allow pass prompt in kwargs & reduce GPU usage for image_inpainting
• Improve video frame interpolation pipeline
• Use package LoadImage for image io in image_quality_assessment_mos
• Remove text2sql_lgesql from nlp requirements
• Remove tensorboard hook as default
• Add model_revision parameter to ImageDetectionDamoyoloTrainer
• Update mgeo finetune test case for rerank
• Add args for asr_infer_pipeline, punc_pipeline, sv_pipeline & modify funasr version
• Add model type check and give easy-to-understand error prompts
• Replace import torchaudio to avoid unnecessary requirements in framework
• Split training and evaluating code for nearfield kws trainer
• Update test image for image deblur
• Add output_dir for asr inferencewhen called
• Support cpu mode for video depth estimation
• Add zhconv to nlp requirements
• Modify the resumable cache path for oss utils
• Support the form of '/to/path/abc.csv' in MsDataset.load() function
• Add UT cases
• Limit pyarrow version

BugFix

• Fix bug in speaker verification infer
• Fix bug when add use_fast for text_ranking
• Fix two ckpt hooks save in the same dir
• Fix the bug that image_color_enhance_pipeline cannot run in CPU environment
• Fix bugs in audio fs, asr & sv demo services
• Fix gpt3 unexpected spaces
• Fix loading checkpoint errors for palm
• Fix data parallel bug for mgeo evaluation
• Fix the checkpoint is incompletely saved with tensor model parallel
• Fix _eval_iters_per_epoch None bug
• Fix damoyolo evaluater load checkpoint not matched
• Fix video matting demo format (mp4v to h264)
• Fix delete model revision
• Fix datasets version incompatible issue
• Fix the compatibility issue of datasets
• Fix postprocessor bugs with batch inference
• Fix asr backward compatibility during inference with tensorflow
• Fix for hand detect finetune
• Fix typos

中文说明

• 语音领域依赖拆分为子领域，减少依赖安装
• 语音唤醒增加返回中文配置支持
• funasr版本升级 & 语音识别、说话人确认、标点预测增加额外参数配置
• 移除基础框架对torchaudio的依赖

English

• separate audio requirements
• kws pipeline returns Chinese charactor by configuration
• add args for asr_infer_pipeline, punc_pipeline, sv_pipeline & modify funasr version
• re-place import torchaudio to avoid unnecessary requirements in framework

中文版本

该版本共新增上架38个模型，其中14个模型支持finetune能力。

模型功能特性说明

• 高性能检测热门应用系列， 基于精度和速度均超越当前经典YOLO系列、面向工业落地的高性能检测框架DAMOYOLO，新增实时口罩检测模型、实时安全帽检测模型、实时人体检测模型、实时香烟检测模型上线，提供开箱即用的高效体验

• 语音识别、语音合成以及语音唤醒可以基于Modelscope Python SDK进行模型finetune

• 语音合成，新增方言模型四川话、广东粤语与上海话，新增俄语与韩语外语模型

  • SambertHifigan语音合成-四川话-通用领域-16k-发音人chuangirl, 方言四川话女声模型

  • SambertHifigan语音合成-广东粤语-通用领域-16k-发音人jiajia， 方言广东话女声模型

  • SambertHifigan语音合成-上海话-通用领域-16k-发音人xiaoda， 方言上海话女声模型

  • SambertHifigan语音合成-俄语-通用领域-16k-发音人masha， 俄语女声模型

  • SambertHifigan语音合成-韩语-通用领域-16k-发音人kyong， 韩语女声模型

• 语音文件后处理

  • 新增英语、德语、菲律宾语、韩语、越南语、日语、俄语、印尼语、葡萄牙语、法语、西班牙等11中语言的文本规整模型

• 图像人脸融合

  • 自动进行人脸区域提取&对齐，并完成面部特征提取，无需额外预处理。

  • 引入3D重建网络对脸型进行拟合迁移，使得融合后的脸型相似度更高。

• 人脸人体

  • GPEN人像增强修复-大分辨率人脸，基于GPEN框架，收集超大分辨率人脸数据训练的1024和2048模型。

• 视觉编辑

  • DDColor图像上色，相比Deoldify等之前方法在色彩丰富度和语义贴合上大幅提升。

  • VFI-RAFT视频插帧，和其它SOTA模型相比，在大运动和重复纹理场景下有较好的插帧效果。

  • DUT-RAFT视频稳像，对多种视频抖动都有稳定的去抖效果，相比原生DUT，能够更好地保持视频清晰度。

• 底层视觉

  • RealBasicVSR视频超分辨率，对于大部分真实场景的视频超分辨率效果良好，对于小部分降质十分严重的情况可能表现不佳。

非兼容性修改

• 文图生成任务输出类型改为多图输出
• 语音合成任务输出数据从output_pcm改为output_wav

新模型列表及快捷访问

贡献组织	模型名称
哔哩哔哩	RealCUGAN图像超分辨率
元语智能	元语功能型对话大模型
封神榜	闻仲-GPT2-110M-中文-v2
封神榜	二郎神-RoBERTa-330M-文本相似度
封神榜	二郎神-RoBERTa-110M-自然语言推理
封神榜	二郎神-RoBERTa-330M-文本相似度
阿里巴巴AAIG	离散对抗训练ViT-H/14-鲁棒图像分类-imagenet1k
阿里云机器学习平台PAI	GPT-MoE中文67亿诗歌生成模型
阿里云机器学习平台PAI	GPT-MoE中文270亿作文生成模型
达摩院	读光-文字检测-单词检测模型-英文-VLPT预训练
达摩院	读光-文档理解-文档理解多模态预训练模型
达摩院	中文StableDiffusion-通用领域
达摩院	DDColor图像上色
达摩院	视频多目标跟踪-行人
达摩院	MaskDINO-SwinL图像实例分割
达摩院	VFI-RAFT视频插帧
达摩院	DUT-RAFT视频稳像
达摩院	RealBasicVSR视频超分辨率
达摩院	GPEN人像增强修复-大分辨率人脸
达摩院	YOLOX-PAI手部检测模型
达摩院	ConvNeXt图像分类-中文-垃圾分类
达摩院	BNext二值化图像分类-英文-通用-small
达摩院	实时口罩检测-通用
达摩院	实时安全帽检测-通用
达摩院	实时香烟检测-通用
达摩院	人脸活体检测模型
达摩院	人脸活体检测模型-IR
达摩院	MGeo多任务多模态地址预训练底座-中文-base
达摩院	MaSTS预训练模型-中文-搜索-CLUE语义匹配-large
达摩院	MaSTS文本相似度-中文-搜索-CLUE语义匹配-large
达摩院	NestedNER命名实体识别-中文-医疗领域-base
达摩院	CoROM文本向量-中文-电商领域-base
达摩院	CoROM语义相关性-中文-电商领域-base
达摩院	全任务零样本学习-mT5分类增强版-中文-base
达摩院	StructBERT情绪分类-中文-七分类-base
达摩院	HiTransUSE用户满意度估计-中文-电商-base
达摩院	UniASR语音识别-中文-通用-8k-实时-pytorch
达摩院	Paraformer语音识别-中文-通用-16k-离线-large-pytorch
达摩院	Paraformer语音识别-中文-通用-16k-离线-large-长音频版
达摩院	RaNER-chunking-英文-large
达摩院	mPLUG-HiTeA-视频问答模型-英文-Base
达摩院	mPLUG-HiTeA-视频描述-英文-Base
达摩院	Mask2Former-R50全景分割
达摩院	图像人脸融合
达摩院	春联生成模型-中文-base
达摩院	GPT-3夸夸机器人-中文-large
达摩院	BART文本纠错-中文-法律领域-large

English Version

Highlight

• Add finetune support for DAMO-YOLO
• Add new real-time mask detection model, real-time helmet detection model, real-time human body detection model, real-time cigarette detection model
• Add finetune support asr, tts and kws model
• Batch inference support for nlp and ofa based multi-modal tasks
• Add high-resolution gpen model for face restoration
• Add DDColor model for image colorization
• Add VFI-RAFT model for video frame interpolation
• Add DUT-RAFT model for video stabilization
• Add RealBasicVSR model for video-super-resolution

Breaking changes

• change output of task text to image to list of images
• change output of task tts from output_pcm to output_wav

Feature

• Add easyrobust-models for image classification
• Video depth estimation support cpu mode
• asr pipeline add output_dir parame
• Add RTS face recognition ood model
• Add image-defrcn-fewshot-detection
• Add hires gpen model
• Add mgeo finetune and pipeline
• Add asr finetune & change inference
• Add quadtree image matching pipeline
• Add finetune for DAMO-YOLO
• Add FLRGB Face Liveness RGB Model
• Add speech separation finetune
• Asr inference: support new models, punctuation, vad, sv
• Add vop retrieval
• Add NAFNet Image Deblurring pipeline and finetune support
• Add megatron bert
• Add panovit-layout-estimation-pipeline
• Add vision middleware
• Add panorama_depth_estimation
• Unify token classfication model output
• Faq support finetune and multilingual
• Support stable diffusion and add DAMO chinese stable diffusion model
• Add cv-bnext-image-classification-pipeline
• Add VFI-RAFT model for video frame interpolation
• Add face changing pipeline
• Add DUT-RAFT model for video stabilization
• Update token_cls default sequence_length: 128 -> 512
• Add structure tasks for ofa: sudoku & text2sql
• Add new ASR model speech_UniASR_asr_2pass-pt-16k-common-vocab1617-tensorflow1-offline and speech_UniASR_asr_2pass-pt-16k-com
• Add model for multiple object tracking in video
• Add ConvNeXt model
• Add ppl metric
• Add image colorization
• Add User satisfaction estimation pipeline
• OFA finetune support configuration file
• Add vldoc model
• Add space-t trainer for finetune
• Add speech separation pipeline
• Add cv_casmvs_multi-view-depth-esimation_general
• Add finetune support for mask2former
• Add domain specific object detection models
• Add maskdino model
• Add FLIR Face Liveness Model
• Add finetune support for kws nearfield
• Add cv_pointnet2_sceneflow-estimation_general
• Add HiTeA model for VideoQA and Caption
• Add GPT-2 model
• Add real-time human detection model
• Add video depth estimation pipeline
• Add ocr-detection-vlpt-pipeline
• Add RealBasicVSR model for video-super-resolution
• Add image skychange model
• Add support for cv_rdevos_video-object-segmentation
• Support kantts infer and finetune
• Add gpt-moe model
• Add hand detection

Improvements

• Text-error-correction support batch inference
• Add beam search and pair finetune for GPT-3
• Optimize ast_index logic
• Refactor msdataset modules
• Save a video with h264 vcodec for video_super_resolution
• Enhance interface standard and refactor card_detection, face detection, tinynas object detection and image classification pipeline
• Audio pipeline Support byte input feature and refine fp implementations
• Remove opencv-python from framework requirements and remove easynlp from nlp default requirements
• GPT-3 model supports batch input
• Batch inference for all ofa models
• AST scanner prebuilt in whl to speed up import process

BugFix

• Fix best ckpt saver not actually save best ckpt error
• Fix logger file hanlder problem
• Fix missing self. (#61)
• Fix: hub test suites can not parallel
• Fix loading custom cv data error (#59)
• Fix saved checkpoint can't run with pipeline for gpt3
• Fix check video type cv2.VideoCapture and add unittest
• Fix a bug for plug inference
• Fix memory leak bug in eval for movie scene segmentation
• Fix useragent string and a trainer invokedby
• Fix: statistics header not correct set
• Fix timeout issue for uni-fold list_oss_objects api
• Fix card-detection model unregistered error and fix log warning
• Fix multimer input for science/protein_structure
• Fix demo service && copy license for cv/language_guided_video_summarization
• Fix file directory create error in ddp training

Highlight

• Add Wenet model #35
• Add code generation and code translation from ZHIPU #33
• Hub support retry and continue-download after error
• Add five finetune tasks for ofa
• Add damoyolo-t and damoyolo-m
• Add dpm-solver for diffusion models
• Add image depth estimation pipeline
• Add en-zh en-es es-en base translation models
• Add GPT-3 tensor parallel finetuning

Feature

• Add five finetune tasks for ofa
• Add synonym for table question answering
• Add jupyter lab plugin in docker
• Add language_guided_video_summarization pipeline
• Add nlp/addr/structure and update token classificaiton related method
• Add damoyolo-t and damoyolo-m
• Add Wenet model #35
• Add code generation and code translation from ZHIPU #33
• Add camouflaged-detection
• Support batch inference in pipeline for some models
• Add table recognition task
• Add dpm-solver for diffusion models
• Ofa add asr task
• Add features for alimeeting competition dataset
• Add funasr based asr inference
• Add extractive-summarization and topic-segmentation
• Add image depth estimation pipeline
• Add en-zh en-es es-en base translation models
• Add gpt-moe model and pipeline
• Action-detection model predownload video before inference
• Add finetune for cv/language_guided_video_summarization
• Add plug finetune and pretrained model
• Support license plate detection
• Add nextvit-small_image-classification_Dailylife-labels model
• Add support for UniTE
• Add video human matting task
• Add LSTMCRFForWordSegmentation
• Add face mask model
• support new asr paraformer and conformer model
• Add GPT-3 tensor parallel finetuning
• Update image-portait-enhancement trainer
• Add FairFace face attribute model
• Add facial landmark confidence model

Improvements

• Hub support retry and continue-download after error
• Refactor NLP and fix some user feedbacks
• Speed up the ast indexing during editing
• Add tensorboard hook for visualization
• reduce the GPU usage of dialog trianer
• substitute face detection model in skin_retouching_pipeline.py
• update git-lfs install instruction

BugFix

• fix output video path when person detect failed For 3d_body_keypoints
• Fix lazy importing problem in text classification pipeline
• Fix bug for distributed inference of gpt3
• Fix bug for mplug evaluation
• token preprocess bug fix
• fix file encoding problem in windows
• fix deadlock when setting the thread number up to 90 for kws model
• fix bug in token classification postprecessor
• fix: torch.concat compatibility with torch1.8
• fix log print and extensions issue for datasets==2.5.2
• fix interpolate value error for vitadapter semantic segmentation
• nlp csanmt translation fix finetuning bug
• Fix a bug that the logging file cannot save the correct lr, which is zero instead
• fix bug of tableQA on gpu
• Fix bug for text generation task model
• fix download file timeout too short