# File System
This is the File System Module, designed to handle file transfer functionalities.
## Overview
The component currently supports three types of IO Handler:
1. scepter.modules.utils.file_clients.AliyunOssFs
2. scepter.modules.utils.file_clients.LocalFs
3. scepter.modules.utils.file_clients.HttpFs
## Basic Usage
```python
from scepter.modules.utils.file_system import FS
from scepter.modules.utils.config import Config
fs_cfg = Config(load=False, cfg_dict={
"NAME": "AliyunOssFs",
# ENDPOINT DESCRIPTION: the oss endpoint TYPE: str default: ''
"ENDPOINT": "xxxxx",
# BUCKET DESCRIPTION: the oss bucket TYPE: str default: ''
"BUCKET": "xxxxx",
# OSS_AK DESCRIPTION: the oss ak TYPE: str default: ''
"OSS_AK": "xxxxx",
# OSS_SK DESCRIPTION: the oss sk TYPE: str default: ''
"OSS_SK": "xxxxx",
# TEMP_DIR DESCRIPTION: default is None, means using system cache dir and auto remove! If you set dir, the data will be saved in this temp dir without autoremoving default. TYPE: NoneType default: None
"TEMP_DIR": "cache",
# AUTO_CLEAN DESCRIPTION: when TEMP_DIR is not None, if you set AUTO_CLEAN to True, the data will be clean automatics. TYPE: bool default: False
"AUTO_CLEAN": False
})
fs_prefix = FS.init_fs_client(fs_cfg, logger=None)
with FS.get_from("xxxxx", wait_finish=True) as local_object:
# do sth. using local_object here.
# Download multiple files at once.
generator = FS.get_batch_objects_from(["xxx", "xxxx"])
for local_path in generator:
print(local_path)
```
## **scepter.modules.utils.file_system.FileSystem**
By building various File IO Handlers, it supports read and write operations for different types of files.
### function **\_\_init\_\_**
()
**Parameters**
### function **init_fs_client**
( cfg: *scepter.modules.utils.config.Config* = None, logger = None ) -> str
The fs_client is instantiated through the cfg parameter and stored in the self._prefix_to_clients attribute, allowing access to the corresponding fs_client via the prefix.
**Parameters**
- **cfg** —— The Config used to build fs_client. If None, use LocalFs as default.
- **logger** —— Instantiated Logger to print or save log.
**Returns**
- *str* —— The prefix of instantiated fs_client
### function **get_fs_client**
( target_path: *str*, safe: *bool* = False )
Retrieve the corresponding fs_client based on the prefix of the target_path.
**Parameters**
- **target_path** —— Target file path.
- **safe** —— In safe mode, return a copy of the client; otherwise, return the client itself.
**Returns**
- *BaseFs* —— Instantiated fs_client.
### function **get_from**
( target_path: *str*, local_path: *str* = None, wait_finish: *bool* = False ) -> str
Download remote files to the local system.
**Parameters**
- **target_path** —— Remote file path.
- **local_path** —— Local file path; if None, use the cache path.
- **wait_finish** —— if True, only the card 0 of each machine will download the data, and the other cards will wait for the download by card 0 to finish.
**Returns**
- *str* —— Local save file path.
### function **get_dir_to_local_dir**
( target_path: *str*, local_path: *str* = None, wait_finish: *bool* = False, timeout: *int* = 3600, worker_id: *int* = 0 ) -> str
Download a folder from a remote path to the local system.
**Parameters**
- **target_path** —— Remote folder path.
- **local_path** —— Local folder path; if None, use the cache path.
- **wait_finish** —— if True，only the 0-card of each machine downloads the data, while the other cards wait for the 0-card to finish downloading.
- **timeout** —— Download timeout duration.
- **worker_id** —— Deprecated
**Returns**
- *str* —— 本地文件夹路径
### function **get_object**
(target_path: *str*) -> bytes
Read a remote file into memory
**Parameters**
- **target_path** — Target file path
**Returns**
- *bytes* — Binary data of the target file
### function **put_object**
(local_data: *bytes*, target_path: *str*) -> bool
Upload a data stream to a specified file
**Parameters**
- **local_data** — Local data stream
- **target_path** — Target file path
**Returns**
- *bool* — Whether the upload was successful
### function **delete_object**
(target_path: *str*) -> bool
Delete the target file
**Parameters**
- **target_path** — Target file path
**Returns**
- *bool* — Whether the deletion was successful
### function **get_batch_objects_from**
(target_path_list: *list[str]*, wait_finish: *bool*) -> Iterator[str]
Batch download files
**Parameters**
- **target_path_list** — List of files to download
**Returns**
- *Iterator[str]* — Iterator for local file paths
-
### function **put_batch_objects_to**
(local_path_list: *list[str]*, target_path_list: *list[str]*, wait_finish: *bool*) -> Iterator[tuple[str, str]]
Batch upload files
**Parameters**
- **local_path_list** — List of files to upload
- **target_path_list** — List of target file paths
**Returns**
- *Iterator[tuple[str, str]]* — Returns pairs of local file and target file paths
### function **get_object_stream**
(target_path: *str*, start: *int*, size: *int*, delimiter: *str*) -> bytes, int
Batch upload files
**Parameters**
- **target_path** — Target file
- **start** — Starting character position of the target stream
- **size** — Size of the target stream starting from the character position
- **delimiter** — Delimiter character for the end of the target stream
**Returns**
- *bytes, int* — Returns the data stream bytes and the end character position
### function **get_object_chunk_list**
(target_path: *str*, chunk_num: *int* = 1, delimiter: *str* = None) -> list[bytes]
Get a remote file and divide it into chunks
**Parameters**
- **target_path** — Target file path
-**chunk_num** — Number of chunks
- **delimiter** — Delimiter to ensure the data downloaded is a complete record and not truncated in the middle
**Returns**
- *list[bytes]* — Chunked data
### function **get_url**
(target_path: *str*, set_public=False, lifecycle: *int* = 360000) -> str
Get the URL of a remote file (only supports AliyunOssFs)
**Parameters**
- **target_path** — Target file path
- **lifecycle** — Valid duration
- **set_public** — Whether to provide a public link
**Returns**
- *str* — URL of the target file
### function **put_to**
(target_path: *str*)
Supports uploading a local file to a remote path
**Parameters**
- **target_path** — Remote file path
**Returns**
- **None**
```python
# Used as a context manager
with FS.put_to(target_path) as local_path:
# some operations on local_path.
```
### function **put_object_from_local_file**
(local_path: *str*, target_path: *str*) -> bool
Push a local file to a remote path
**Parameters**
- **local_path** — Local file path
- **target_path** — Remote file path
**Returns**
- *bool* — Whether the upload was successful
### function **put_dir_from_local_dir**
(local_dir: *str*, target_dir: *str*) -> bool
Push a local directory to a remote path
**Parameters**
- **local_dir** — Local directory path
- **target_dir** — Remote directory path
**Returns**
- *bool* — Whether the upload was successful
### function **add_target_local_map**
(target_dir: *str*, local_dir: *str*) -> None
Save the mapping relationship between the remote directory path and the local directory path as key-value pairs to self._target_local_mapper
**Parameters**
- **target_dir** — Remote directory path
- **local_dir** — Local directory path
**Returns**
- *None*
### function **make_dir**
(target_dir: *str*) -> bool
Create a remote directory
**Parameters**
- **target_dir** — Remote directory path
**Returns**
- *bool* — Whether the creation was successful
### function **exists**
(target_path: *str*) -> bool
Check if the target path exists
**Parameters**
- **target_path** — Remote file path
**Returns**
- *bool* — Whether it exists
### function **map_to_local**
(target_path: *str*) -> str, bool
Map the remote file path to a local path
**Parameters**
- **target_path** — Remote file path
**Returns**
- *str* — Local file path
- *bool* — Whether the local file is a temporary file
### function **walk_dir**
(target_dir: *str*, recurse=True) -> Iterator
Get the file list under the remote directory
**Parameters**
- **target_dir** — Remote directory path
- **recurse** — Whether to traverse subdirectories, default is True
**Returns**
- *Iterator* — List of subfile paths
### function **is_local_client**
(target_path: *str*) -> bool
Determine if the target file client is LocalFs
**Parameters**
- **target_path** — Target file path
**Returns**
- *bool* — Whether the client is LocalFs
### function **size**
(target_path: *str*) -> int
Determine the size of the target file
**Parameters**
- **target_path** — Target file path
**Returns**
- *int* — Size of the target file
### function **isfile**
(target_path: *str*) -> bool
Determine if the target path is an object
**Parameters**
- **target_path** — Target file path
**Returns**
- *bool* — Whether the target path is an object
### function **isdir**
(target_path: *str*) -> bool
Determine if the target path is a directory
**Parameters**
- **target_path** — Target file path
**Returns**
- *bool* — Whether the target path is a directory
## **scepter.modules.utils.file_clients.AliyunOssFs**
```yaml
NAME: AliyunOssFs
TEMP_DIR: None
AUTO_CLEAN: False
ENDPOINT:
BUCKET:
OSS_AK:
OSS_SK:
PREFIX: ""
WRITABLE: True
CHECK_WRITABLE: False
RETRY_TIMES: 10
```
## **scepter.modules.utils.file_clients.LocalFs**
```yaml
NAME: LocalFs
TEMP_DIR: None
AUTO_CLEAN: False
```
## **scepter.modules.utils.file_clients.HttpFs**
```yaml
NAME: HttpFs
TEMP_DIR: None
AUTO_CLEAN: False
RETRY_TIMES: 10
```