# libtorch_tokenizer

**Repository Path**: RapidAI/libtorch_tokenizer

## Basic Information

- **Project Name**: libtorch_tokenizer
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2023-03-22
- **Last Updated**: 2023-03-22

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# Huggingface Transformers Tokenizer in C++

A tokenizer is in charge of preparing the inputs for a model. 

The tokenizer can tokenize Chinese-English bilingual in Linux.

This project mainly solves some Chinese character encoding problems.

Requirements

- [x] Boost
  
 ### C++ unicode support 
- http://github.com/ufal/unilib